Intelligent Imaging Solutions GmbH (IIS) is a company offering software solutions in the field of computational photography. Over the past few years, it has successfully marketed software solutions for end-consumers in the photography market. The software solutions were highly specialized and the first of their kind. These solutions received a lot of attention and positive feedback from both customers and the media.
The products are aimed towards consumers in the digital-camera market. Even though cameras and lenses in this market can cost several thousand euros, the full potential of the technology has not been fully unlocked so far. The feasibility study completed under the Horizon2020 scheme had two goals to change this:
1) To demonstrate that modern computers with powerful graphic cards (GPUs) can be used to process and enhance high-resolution aerial photos within a reasonable time.
2) To evaluate the market potential for a software solution capable of correcting complex optical aberrations in the aerial-camera market.
Camera lenses are a critical component of optical-imaging systems, and lens imperfections compromise image quality. Manufacturers of photographic lenses attempt to minimize optical aberrations by combining several glass elements in so-called compound lenses comprising as many as fifteen elements or more. As a result, high-grade lenses are probably the most expensive components of high-end camera systems today and specialized lenses for aerial photography can cost tens of thousands of Euros. Optical aberrations are inevitable and the design of a lens is always a trade-off between various parameters, including price. In fact, it is impossible to make a perfect lens, and it is very expensive to make a close-to-perfect lens. This has not been a big concern over the past decades in photography as the optical quality of lenses easily outperformed the sensor / film resolution. However, this has changed recently due to the ever increasing resolution of modern camera sensors. Optical aberrations have now become a problem for image quality—and it becomes increasingly expensive to correct optical aberrations, particularly for challenging lighting situations. If one wanted to improve the optical performance of lenses for high-resolution aerial cameras further, the costs would increase by 200–500 percent which would make these cameras economically unfeasible.
A much cheaper solution can be found within the new field of computational photography. IIS is currently the only company capable of correcting spatially-varying complex optical aberrations. This means that IIS can significantly improve the sharpness of images from high-resolution sensors that suffer from a decrease in sharpness towards the edges. The process of reverting image degradations stemming from imperfect optics by computational means is known as image deconvolution.
There are several reasons why correcting optical aberrations by the means of image deconvolution have not been widely used so far:
• The optical aberrations need to be known and described in a mathematically complex way. Determining these aberrations is very labor intensive and they vary significantly for different aperture and focus settings. Lens-to-lens variation is another problem making it almost impossible to leverage the technology on a larger scale particularly in the consumer sector.
• Reversing optical aberrations is computationally very expensive, especially for large images. The information of several hundreds (or even thousands) of pixels in the neighboring region has to be taken into account in order to recover a single pixel. Recovering an image of 60–200 MP in size is computationally challenging and only possible on high-performance computers. So far, the computation time for a single image was in the order of tens of minutes which is too long for the field of aerial photography, where typically hundreds to thousands of images are taken during a single flight.
• It is a relatively new field with very few companies in the market. Development is very expensive and very risky.
For the field of aerial photography, measuring lenses to determine the optical aberrations is feasible as the economic costs for determining the optical aberrations is small compared to the overall system costs. An aerial camera system is generally priced upwards of 30,000 EUR. The core challenge with aerial-camera systems lies with the fast correction of a large set of images. It is not unusual to capture several terabytes of data within one image acquisition session. The ability to correct optical aberrations and improve image quality in aerial photography, therefore, largely depends on the processing speed. So far, even highly optimized and robust routines were only able to process about 0.1 MP/sec on a modern multi-core computer. A single 80 MP image would, therefore, have taken close to 15 minutes—with thousands of images in the queue waiting to be processed. This is not a feasible solution. Processing large image sets in computer clusters is not possible either due to the large amount of data that has to be transferred. The only possibility to improve image quality “locally” and within a reasonable amount of time appears to be leveraging the computational power of GPUs. In the scope of the feasibility study, the possibilities of speeding up the correction of large images by means of deconvolution on a GPU and the possibilities in the market are evaluated.
To this end, IIS ported computationally intensive routines to GPU code and optimized it for large images. The particular challenge with this task was not obvious at first. While it is well known that certain operations such as Fourier transformations (FFTs) can be speeded up significantly on a GPU, the design and architecture of the routines differ tremendously on the GPU architecture. Unless the concept is actually implemented and tested on a GPU, it is unclear how big the performance improvements are. Initial tests showed a speed increase of only 30 percent on a modern GPU compared to the CPU version. This increase fell somewhat short of expectations. Redesigning the algorithms and the architecture allowed IIS to achieve a speed increase of 80 percent on a modern GPU compared to highly optimized CPU algorithms. An important prerequisite was to achieve the improvement on a high-end GPU with a per unit cost of less than 1,000 EUR. The prototype, therefore, was able to demonstrate that with GPU acceleration, it is possible to process even large images of 60+MP in about one minute locally (multi-GPU, additional optimization)—an important prerequisite to making this technology feasible at all.
As a result, the first goal of the feasibility study was achieved: IIS was able to demonstrate that the technology capable of reversing complex optical aberrations in high-resolution camera systems can be integrated in the image processing workflow on a modern workstation and processing times are within an acceptable range.
Aerial cameras are traditionally developed and marketed by “hardware companies”. In this industry, software is not widely acknowledged as an important piece of the processing pipeline. Customers, as well as manufacturers, prefer a “hardware” solution over a “software” solution in most cases—even if the software solution is cheaper. Another big challenge in the feasibility study was understanding the willingness of the market to adopt a solution that would allow “imperfections from hardware” to be corrected through software. However, as the costs for improving image quality through hardware (optics) only keep increasing to unreasonable levels, camera manufacturers were actually keen on learning about the potential of computational photography. As this is a highly specialized market with only a few players dominating the market with complex aftersales support and customers spread across vast geographical regions, it appears that marketing such a solution directly to the end customer is not feasible. It is important to integrate the solution seamlessly into the post-processing workflow which is only possible through close collaboration with the camera manufacturers themselves.
