Periodic Reporting for period 1 - Virtual Optics (Virtual Optics - A software revolution in the optical industry)
Reporting period: 2015-09-01 to 2016-04-30
Intelligent Imaging Solutions GmbH (IIS) is a company offering software solutions in the field of computational photography. Over the past few years, it has successfully marketed software solutions for end-consumers in the photography market. The software solutions were highly specialized and the first of their kind. These solutions received a lot of attention and positive feedback from both customers and the media.
The products are aimed towards consumers in the digital-camera market. Even though cameras and lenses in this market can cost several thousand euros, the full potential of the technology has not been fully unlocked so far. The feasibility study completed under the Horizon2020 scheme had two goals to change this:
1) To demonstrate that modern computers with powerful graphic cards (GPUs) can be used to process and enhance high-resolution aerial photos within a reasonable time.
2) To evaluate the market potential for a software solution capable of correcting complex optical aberrations in the aerial-camera market.
Camera lenses are a critical component of optical-imaging systems, and lens imperfections compromise image quality. Manufacturers of photographic lenses attempt to minimize optical aberrations by combining several glass elements in so-called compound lenses comprising as many as fifteen elements or more. As a result, high-grade lenses are probably the most expensive components of high-end camera systems today and specialized lenses for aerial photography can cost tens of thousands of Euros. Optical aberrations are inevitable and the design of a lens is always a trade-off between various parameters, including price. In fact, it is impossible to make a perfect lens, and it is very expensive to make a close-to-perfect lens. This has not been a big concern over the past decades in photography as the optical quality of lenses easily outperformed the sensor / film resolution. However, this has changed recently due to the ever increasing resolution of modern camera sensors. Optical aberrations have now become a problem for image quality—and it becomes increasingly expensive to correct optical aberrations, particularly for challenging lighting situations. If one wanted to improve the optical performance of lenses for high-resolution aerial cameras further, the costs would increase by 200–500 percent which would make these cameras economically unfeasible.
A much cheaper solution can be found within the new field of computational photography. IIS is currently the only company capable of correcting spatially-varying complex optical aberrations. This means that IIS can significantly improve the sharpness of images from high-resolution sensors that suffer from a decrease in sharpness towards the edges. The process of reverting image degradations stemming from imperfect optics by computational means is known as image deconvolution.
There are several reasons why correcting optical aberrations by the means of image deconvolution have not been widely used so far:
• The optical aberrations need to be known and described in a mathematically complex way. Determining these aberrations is very labor intensive and they vary significantly for different aperture and focus settings. Lens-to-lens variation is another problem making it almost impossible to leverage the technology on a larger scale particularly in the consumer sector.
• Reversing optical aberrations is computationally very expensive, especially for large images. The information of several hundreds (or even thousands) of pixels in the neighboring region has to be taken into account in order to recover a single pixel. Recovering an image of 60–200 MP in size is computationally challenging and only possible on high-performance computers. So far, the computation time for a single image was in the order of tens of minutes which is too long for the field of aerial photography, where typically hundreds to thousands of images are taken during a single flight.
• It is a relatively new field with very few companies in the market. Development is very expensive and very risky.
For the field of aerial photography, measuring lenses to determine the optical aberrations is feasible as the economic costs for determining the optical aberrations is small compared to the overall system costs. An aerial camera system is generally priced upwards of 30,000 EUR. The core challenge with aerial-camera systems lies with the fast correction of a large set of images. It is not unusual to capture several terabytes of data within one image acquisition session. The ability to correct optical aberrations and improve image quality in aerial photography, therefore, largely depends on the processing speed. So far, even highly optimized and robust routines were only able to process about 0.1 MP/sec on a modern multi-core computer. A single 80 MP image would, therefore, have taken close to 15 minutes—with thousands of images in the queue waiting to be processed. This is not a feasible solution. Processing large image sets in computer clusters is not possible either due to the large amount of data that has to be transferred. The only possibility to improve image quality “locally” and within a reasonable amount of time appears to be leveraging the computational power of GPUs. In the scope of the feasibility study, the possibilities of speeding up the correction of large images by means of deconvolution on a GPU and the possibilities in the market are evaluated.
To this end, IIS ported computationally intensive routines to GPU code and optimized it for large images. The particular challenge with this task was not obvious at first. While it is well known that certain operations such as Fourier transformations (FFTs) can be speeded up significantly on a GPU, the design and architecture of the routines differ tremendously on the GPU architecture. Unless the concept is actually implemented and tested on a GPU, it is unclear how big the performance improvements are. Initial tests showed a speed increase of only 30 percent on a modern GPU compared to the CPU version. This increase fell somewhat short of expectations. Redesigning the algorithms and the architecture allowed IIS to achieve a speed increase of 80 percent on a modern GPU compared to highly optimized CPU algorithms. An important prerequisite was to achieve the improvement on a high-end GPU with a per unit cost of less than 1,000 EUR. The prototype, therefore, was able to demonstrate that with GPU acceleration, it is possible to process even large images of 60+MP in about one minute locally (multi-GPU, additional optimization)—an important prerequisite to making this technology feasible at all.
As a result, the first goal of the feasibility study was achieved: IIS was able to demonstrate that the technology capable of reversing complex optical aberrations in high-resolution camera systems can be integrated in the image processing workflow on a modern workstation and processing times are within an acceptable range.
Aerial cameras are traditionally developed and marketed by “hardware companies”. In this industry, software is not widely acknowledged as an important piece of the processing pipeline. Customers, as well as manufacturers, prefer a “hardware” solution over a “software” solution in most cases—even if the software solution is cheaper. Another big challenge in the feasibility study was understanding the willingness of the market to adopt a solution that would allow “imperfections from hardware” to be corrected through software. However, as the costs for improving image quality through hardware (optics) only keep increasing to unreasonable levels, camera manufacturers were actually keen on learning about the potential of computational photography. As this is a highly specialized market with only a few players dominating the market with complex aftersales support and customers spread across vast geographical regions, it appears that marketing such a solution directly to the end customer is not feasible. It is important to integrate the solution seamlessly into the post-processing workflow which is only possible through close collaboration with the camera manufacturers themselves.
The products are aimed towards consumers in the digital-camera market. Even though cameras and lenses in this market can cost several thousand euros, the full potential of the technology has not been fully unlocked so far. The feasibility study completed under the Horizon2020 scheme had two goals to change this:
1) To demonstrate that modern computers with powerful graphic cards (GPUs) can be used to process and enhance high-resolution aerial photos within a reasonable time.
2) To evaluate the market potential for a software solution capable of correcting complex optical aberrations in the aerial-camera market.
Camera lenses are a critical component of optical-imaging systems, and lens imperfections compromise image quality. Manufacturers of photographic lenses attempt to minimize optical aberrations by combining several glass elements in so-called compound lenses comprising as many as fifteen elements or more. As a result, high-grade lenses are probably the most expensive components of high-end camera systems today and specialized lenses for aerial photography can cost tens of thousands of Euros. Optical aberrations are inevitable and the design of a lens is always a trade-off between various parameters, including price. In fact, it is impossible to make a perfect lens, and it is very expensive to make a close-to-perfect lens. This has not been a big concern over the past decades in photography as the optical quality of lenses easily outperformed the sensor / film resolution. However, this has changed recently due to the ever increasing resolution of modern camera sensors. Optical aberrations have now become a problem for image quality—and it becomes increasingly expensive to correct optical aberrations, particularly for challenging lighting situations. If one wanted to improve the optical performance of lenses for high-resolution aerial cameras further, the costs would increase by 200–500 percent which would make these cameras economically unfeasible.
A much cheaper solution can be found within the new field of computational photography. IIS is currently the only company capable of correcting spatially-varying complex optical aberrations. This means that IIS can significantly improve the sharpness of images from high-resolution sensors that suffer from a decrease in sharpness towards the edges. The process of reverting image degradations stemming from imperfect optics by computational means is known as image deconvolution.
There are several reasons why correcting optical aberrations by the means of image deconvolution have not been widely used so far:
• The optical aberrations need to be known and described in a mathematically complex way. Determining these aberrations is very labor intensive and they vary significantly for different aperture and focus settings. Lens-to-lens variation is another problem making it almost impossible to leverage the technology on a larger scale particularly in the consumer sector.
• Reversing optical aberrations is computationally very expensive, especially for large images. The information of several hundreds (or even thousands) of pixels in the neighboring region has to be taken into account in order to recover a single pixel. Recovering an image of 60–200 MP in size is computationally challenging and only possible on high-performance computers. So far, the computation time for a single image was in the order of tens of minutes which is too long for the field of aerial photography, where typically hundreds to thousands of images are taken during a single flight.
• It is a relatively new field with very few companies in the market. Development is very expensive and very risky.
For the field of aerial photography, measuring lenses to determine the optical aberrations is feasible as the economic costs for determining the optical aberrations is small compared to the overall system costs. An aerial camera system is generally priced upwards of 30,000 EUR. The core challenge with aerial-camera systems lies with the fast correction of a large set of images. It is not unusual to capture several terabytes of data within one image acquisition session. The ability to correct optical aberrations and improve image quality in aerial photography, therefore, largely depends on the processing speed. So far, even highly optimized and robust routines were only able to process about 0.1 MP/sec on a modern multi-core computer. A single 80 MP image would, therefore, have taken close to 15 minutes—with thousands of images in the queue waiting to be processed. This is not a feasible solution. Processing large image sets in computer clusters is not possible either due to the large amount of data that has to be transferred. The only possibility to improve image quality “locally” and within a reasonable amount of time appears to be leveraging the computational power of GPUs. In the scope of the feasibility study, the possibilities of speeding up the correction of large images by means of deconvolution on a GPU and the possibilities in the market are evaluated.
To this end, IIS ported computationally intensive routines to GPU code and optimized it for large images. The particular challenge with this task was not obvious at first. While it is well known that certain operations such as Fourier transformations (FFTs) can be speeded up significantly on a GPU, the design and architecture of the routines differ tremendously on the GPU architecture. Unless the concept is actually implemented and tested on a GPU, it is unclear how big the performance improvements are. Initial tests showed a speed increase of only 30 percent on a modern GPU compared to the CPU version. This increase fell somewhat short of expectations. Redesigning the algorithms and the architecture allowed IIS to achieve a speed increase of 80 percent on a modern GPU compared to highly optimized CPU algorithms. An important prerequisite was to achieve the improvement on a high-end GPU with a per unit cost of less than 1,000 EUR. The prototype, therefore, was able to demonstrate that with GPU acceleration, it is possible to process even large images of 60+MP in about one minute locally (multi-GPU, additional optimization)—an important prerequisite to making this technology feasible at all.
As a result, the first goal of the feasibility study was achieved: IIS was able to demonstrate that the technology capable of reversing complex optical aberrations in high-resolution camera systems can be integrated in the image processing workflow on a modern workstation and processing times are within an acceptable range.
Aerial cameras are traditionally developed and marketed by “hardware companies”. In this industry, software is not widely acknowledged as an important piece of the processing pipeline. Customers, as well as manufacturers, prefer a “hardware” solution over a “software” solution in most cases—even if the software solution is cheaper. Another big challenge in the feasibility study was understanding the willingness of the market to adopt a solution that would allow “imperfections from hardware” to be corrected through software. However, as the costs for improving image quality through hardware (optics) only keep increasing to unreasonable levels, camera manufacturers were actually keen on learning about the potential of computational photography. As this is a highly specialized market with only a few players dominating the market with complex aftersales support and customers spread across vast geographical regions, it appears that marketing such a solution directly to the end customer is not feasible. It is important to integrate the solution seamlessly into the post-processing workflow which is only possible through close collaboration with the camera manufacturers themselves.
The feasibility study and the evaluation of the economic potential comprised several work packages:
• Optimize the critical parts of the base code (technology) for GPUs and thus achieve a significant increase in speed:
o The core algorithm for reversing optical aberrations was successfully (in parts) ported to GPU code. The speed increase was close to 80 percent—significantly better than expected. The prototype, therefore, proved the feasibility of the technology in the image-processing pipeline with respect to processing times.
o The initial idea to determine the optical aberrations “blindly” (without calibration data) was abandoned as the improvements were not as good as expected and retrieving calibration data from the lens can be performed in a cost efficient way for a small number of lenses in the aerial-imaging sector. For the mass consumer market this is not possible as the costs for determining the measurement data is higher than the cost of producing a consumer lens.
o The objective to optimize the crucial parts of the GPU code was, therefore, successfully fulfilled. The technological approach was adopted to a “non-blind” correction of the optical aberrations after some coordination and initial tests with an aerial camera manufacturer.
• Evaluate the market potential for aerial-camera systems
o The market potential that was initially envisioned for this technology was confirmed.
o The best strategy to serve the market was through manufacturers of aerial-camera systems. The business model, therefore, focuses on licensing the technology.
• Finalize market potential and determine pricing of technology to end customers (B2B) and licensing partners:
o Possible partners were identified and the technology was demonstrated
The feasibility study confirmed the market potential, pricing, and the technology. Several adjustments to the “go to market” strategy as well as to the technological approach were made throughout the feasibility study.
• Optimize the critical parts of the base code (technology) for GPUs and thus achieve a significant increase in speed:
o The core algorithm for reversing optical aberrations was successfully (in parts) ported to GPU code. The speed increase was close to 80 percent—significantly better than expected. The prototype, therefore, proved the feasibility of the technology in the image-processing pipeline with respect to processing times.
o The initial idea to determine the optical aberrations “blindly” (without calibration data) was abandoned as the improvements were not as good as expected and retrieving calibration data from the lens can be performed in a cost efficient way for a small number of lenses in the aerial-imaging sector. For the mass consumer market this is not possible as the costs for determining the measurement data is higher than the cost of producing a consumer lens.
o The objective to optimize the crucial parts of the GPU code was, therefore, successfully fulfilled. The technological approach was adopted to a “non-blind” correction of the optical aberrations after some coordination and initial tests with an aerial camera manufacturer.
• Evaluate the market potential for aerial-camera systems
o The market potential that was initially envisioned for this technology was confirmed.
o The best strategy to serve the market was through manufacturers of aerial-camera systems. The business model, therefore, focuses on licensing the technology.
• Finalize market potential and determine pricing of technology to end customers (B2B) and licensing partners:
o Possible partners were identified and the technology was demonstrated
The feasibility study confirmed the market potential, pricing, and the technology. Several adjustments to the “go to market” strategy as well as to the technological approach were made throughout the feasibility study.
The feasibility study confirmed that IIS is currently the technological market leader for correcting optical aberrations in high-resolution cameras. It was demonstrated that the algorithms can be speeded up significantly for large images leveraging GPUs—thus opening up new possibilities that were not thought of before.
The potential impact could be far reaching as the technology can be considered disruptive in the market for aerial cameras. However, to exploit the full potential, a strong market player has to be won over. Integrating the algorithms in the image-processing pipeline could further strengthen Europe’s position as the leader for aerial cameras and protect jobs directly and indirectly against competition from abroad. Over the past years, technological advancements in optics from Chinese and Korean companies posed a significant threat to the existing European players. Whereas the optical and chip industry has mainly moved to the East, software development for crucial components is still done in Europe. If the share of value creation achieved by means of software is increased, so is the sustainable competitive advantage and ultimately job security in Europe.
The potential impact could be far reaching as the technology can be considered disruptive in the market for aerial cameras. However, to exploit the full potential, a strong market player has to be won over. Integrating the algorithms in the image-processing pipeline could further strengthen Europe’s position as the leader for aerial cameras and protect jobs directly and indirectly against competition from abroad. Over the past years, technological advancements in optics from Chinese and Korean companies posed a significant threat to the existing European players. Whereas the optical and chip industry has mainly moved to the East, software development for crucial components is still done in Europe. If the share of value creation achieved by means of software is increased, so is the sustainable competitive advantage and ultimately job security in Europe.