Enhanced multimedia processing
Visual saliency is of increasing importance to computer processing of high-resolution images and videos. One of the current limitations associated with ultra-high-resolution video is the immenseness and complexity of the associated data. The EU supported the project 'Saliency-aware high-resolution video processing' (SHIVPRO) to develop much-needed improved saliency models. To tackle the issue of complex multi-scale images, the project's spatiotemporal model uses region-based features and statistical measures of saliency to modify saliency within a given scale. It propagates that modification using saliency-directed region merging across scales with flexible incorporation of top-down information. The first step was development of a region-based saliency model exploiting histograms of colour-spatial similarities, and two measures of those histograms — distinctiveness and compactness. Next was detection of the relevant salient features based on the saliency model. Scientists proposed a novel concept, the saliency tree. The image is simplified to generate primitive regions whose saliency measures are compared to global ones. A region merging approach generates the tree in which each leaf node represents a primitive region and each non-leaf node a non-primitive region. A systematic saliency analysis produces a high-quality pixel-wise saliency map. Based on the above models, the team developed their efficient superpixel-based spatiotemporal saliency model. Motion and colour histograms are extracted from the superpixel data. A pixel-level spatial saliency map and a similar temporal one are generated and an adaptive fusion method is used to merge them. The model demonstrated superior performance in saliency detection and human fixation prediction relative to the state of the art. The team also investigated the emerging topic of co-saliency detection or identification of common salient objects in a set of images, and proposed hierarchical segmentation-based co-saliency models to do so. Important work advanced the state of the art of high-efficiency video coding with improved coding efficiency and reduced computational complexity. The ability to extract subjectively important features to simplify processing of huge data sets for business, home and mobile video applications is of key socioeconomic importance. EU-funded scientists have made a major contribution with enhancements in saliency modelling and detection demonstrated to outperform the current state of the art.
Keywords
Multimedia processing, saliency, high-resolution, video content, spatiotemporal
