Final Report Summary - IMASEG3D (Learning to Combine Hierarchical Image Modeling with 2-D Segmentation and 3-D Pose Recovery of Visual Objects)
Project context and objectives
The main goal of this project was the design of an innovative computer vision methodology and system to deal with all stages in a visual classification system. We successfully met several of the goals initially planned for the project and some additional goals. Perhaps the most important contribution of this project is its instrumental importance in the development of at least two new projects within the Instituto Superior Técnico (IST, Portugal), which have been financed by the Fundacao para a Ciencia e Tecnologia (FCT). In fact, the support and work of Dr Carneiro was key to the design of the proposals, successfully conducted the projects with quite significant results for the IST and Portugal.
Project results
In terms of the technical results, Dr Carneiro has proposed a large number of new ideas in the field of computer vision, which have attracted the attention of the foremost researchers in the field. The beginning of the project consisted of a thorough investigation on a new type of image representation that can be widely applicable to several visual classification problems. The innovative proposal is based on state-of-the-art machine learning techniques, and the paper was accepted for publication by the top ranked conference in computer vision (the Institute of electrical and electronics engineers (IEEE) Conference on Computer Vision and Pattern Recognition CVPR) to be presented orally, which is an honour given only to the top 3% of the submitted papers.
Dr Carneiro also investigated a hierarchical classifier based on a new idea in the field of machine learning, called the deep belief network. The success of this idea attracted other groups from IST to develop a joint project with Dr Carneiro in the area of medical image analysis. The deep belief network was used not only for image classification, but also for 2D image segmentation, and the results obtained were outstanding. This project resulted in a solid system which has been extensively evaluated by Portuguese doctors with quite promising results. Moreover, this project has generated a large number of papers published in the top conferences and journals in computer vision and medical imaging. In total, Dr Carneiro published seven conference papers and one journal paper (IEEE Transactions on Image Processing), and he is currently working on a revision for another journal paper to be published at the IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). It is possible that this system can be applied in future implementations of medical image analysis systems to be deployed commercially.
Furthermore, Dr Carneiro proposed an innovative methodology for the classification of visual information based on manifolds representing the link between images in a database. This idea is conceptually similar to the approach used by Google and Yahoo to retrieve web pages on the Internet, but its application in visual classification problems has not been equally successful due to a representation problem. Specifically, web pages can be naturally represented by words, but there is not an equivalent visual word representation that is widely accepted and used throughout the field of computer vision. Using the aforementioned image representation combined with the manifold that links database images in terms of visual appearance, textual and graphical annotation, Dr Carneiro was able to build an impressive visual classification system. This approach helped IST acquire a project on artistic image annotation and classification, which is currently being developed with the National Tile Museum (located in Lisbon, Portugal). This partnership is opening several research areas because of its interdisciplinary nature, and IST is now involved in a couple of European networks that deal with artistic image processing. This successful enterprise will be extremely important not only for IST, but also for Portugal, and we plan to extend and expand this work with European and North American partners with similar research interests. It is important to mention that Dr Carneiro has published six conference papers and is currently working on two journal versions of this work. Also, Dr Carneiro participated actively in the organisation of two workshops about artistic image analysis, one held in 2010 at the National Tile Museum (with the participation of the influential art historian Prof. Benito Navarrete and computer vision scientist Prof. David Lowe), and another one to be held in Florence during the European Conference on Computer Vision in 2012, which is the most important meeting of European computer vision scientists held every two years.
Dr Carneiro was offered a tenure track associate professor position at the University of Adelaide (Australia) during the second half of the project, which interrupted the project in July 2011 (i.e. seven months before its official end). Even with the early termination of his contract, we believe that Dr Carneiro did an outstanding job, and we are quite pleased with the results. In summary, with his help, we have been able to acquire two projects, open new research areas, increase our visibility in the computer vision community (via publication in top conferences and journals in the field and also with the organisation of workshops that attracted world-class researchers), and develop one visual representation and two visual classification systems.
Project website
http://users.isr.ist.utl.pt/~gcarneiro/mariecurie.html(opens in new window)
The main goal of this project was the design of an innovative computer vision methodology and system to deal with all stages in a visual classification system. We successfully met several of the goals initially planned for the project and some additional goals. Perhaps the most important contribution of this project is its instrumental importance in the development of at least two new projects within the Instituto Superior Técnico (IST, Portugal), which have been financed by the Fundacao para a Ciencia e Tecnologia (FCT). In fact, the support and work of Dr Carneiro was key to the design of the proposals, successfully conducted the projects with quite significant results for the IST and Portugal.
Project results
In terms of the technical results, Dr Carneiro has proposed a large number of new ideas in the field of computer vision, which have attracted the attention of the foremost researchers in the field. The beginning of the project consisted of a thorough investigation on a new type of image representation that can be widely applicable to several visual classification problems. The innovative proposal is based on state-of-the-art machine learning techniques, and the paper was accepted for publication by the top ranked conference in computer vision (the Institute of electrical and electronics engineers (IEEE) Conference on Computer Vision and Pattern Recognition CVPR) to be presented orally, which is an honour given only to the top 3% of the submitted papers.
Dr Carneiro also investigated a hierarchical classifier based on a new idea in the field of machine learning, called the deep belief network. The success of this idea attracted other groups from IST to develop a joint project with Dr Carneiro in the area of medical image analysis. The deep belief network was used not only for image classification, but also for 2D image segmentation, and the results obtained were outstanding. This project resulted in a solid system which has been extensively evaluated by Portuguese doctors with quite promising results. Moreover, this project has generated a large number of papers published in the top conferences and journals in computer vision and medical imaging. In total, Dr Carneiro published seven conference papers and one journal paper (IEEE Transactions on Image Processing), and he is currently working on a revision for another journal paper to be published at the IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). It is possible that this system can be applied in future implementations of medical image analysis systems to be deployed commercially.
Furthermore, Dr Carneiro proposed an innovative methodology for the classification of visual information based on manifolds representing the link between images in a database. This idea is conceptually similar to the approach used by Google and Yahoo to retrieve web pages on the Internet, but its application in visual classification problems has not been equally successful due to a representation problem. Specifically, web pages can be naturally represented by words, but there is not an equivalent visual word representation that is widely accepted and used throughout the field of computer vision. Using the aforementioned image representation combined with the manifold that links database images in terms of visual appearance, textual and graphical annotation, Dr Carneiro was able to build an impressive visual classification system. This approach helped IST acquire a project on artistic image annotation and classification, which is currently being developed with the National Tile Museum (located in Lisbon, Portugal). This partnership is opening several research areas because of its interdisciplinary nature, and IST is now involved in a couple of European networks that deal with artistic image processing. This successful enterprise will be extremely important not only for IST, but also for Portugal, and we plan to extend and expand this work with European and North American partners with similar research interests. It is important to mention that Dr Carneiro has published six conference papers and is currently working on two journal versions of this work. Also, Dr Carneiro participated actively in the organisation of two workshops about artistic image analysis, one held in 2010 at the National Tile Museum (with the participation of the influential art historian Prof. Benito Navarrete and computer vision scientist Prof. David Lowe), and another one to be held in Florence during the European Conference on Computer Vision in 2012, which is the most important meeting of European computer vision scientists held every two years.
Dr Carneiro was offered a tenure track associate professor position at the University of Adelaide (Australia) during the second half of the project, which interrupted the project in July 2011 (i.e. seven months before its official end). Even with the early termination of his contract, we believe that Dr Carneiro did an outstanding job, and we are quite pleased with the results. In summary, with his help, we have been able to acquire two projects, open new research areas, increase our visibility in the computer vision community (via publication in top conferences and journals in the field and also with the organisation of workshops that attracted world-class researchers), and develop one visual representation and two visual classification systems.
Project website
http://users.isr.ist.utl.pt/~gcarneiro/mariecurie.html(opens in new window)