Community Research and Development Information Service - CORDIS

H2020

DBRLive Report Summary

Project ID: 673682

Periodic Reporting for period 2 - DBRLive (Accelerating innovative augmented reality broadcast applications through new industrial camera design)

Reporting period: 2016-04-01 to 2017-01-31

Summary of the context and overall objectives of the project

The outcome of the project will be to simplify, miniaturize and scale a key layer of the technology solution, namely the camera used to capture real life images and provide the data required for augmented reality applications such as virtual graphics intentions. The target to simplify the solution has become even more important along with the recent change of Supponor’s business concept, i.e., changing from service company to a technology development and licensing company. The new concept means that the less trained customer staff will take care of field operations. This change also enables faster growth of Supponor’s business. The main technical goals are the support for the future increased resolution (terms 4K and UHD are used in this context) of sports broadcasting, simplified effort in setting up and operating the camera system, and improved near infrared (NIR) imaging performance in terms of sensitivity and dynamic range.

The early calculations demonstrate that approx. 10 fold sensitivity in near infrared (NIE) imaging will be achieved, compared to Supponor’s existing system, and that the dynamic range in NIR imaging will improve 100 fold, from 60 dB to 100 dB. The system will employ a modern Internet protocol in data communication that allows easy distribution of the image information to multiple recipients and lowers the overall cost.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Architecture of the optical system:
Two main system architectures of the new optical system were investigated:
1. A solution with external near infrared (NIR) lenses and NIR cameras in addition to regular TV lens and cameras.
2. The solution resembling the current system architecture: A standard TV lens, followed by beam splitter to separate the NIR bands, and a relay lens with iris to re-focus the output image from beam splitter for the TV camera.

Alternative 1 proved to be too bulky because of multiple lenses and cameras, even though special distortive lenses would be employed. Yet there would exist the parallax problem that would be hard to manage in cases where, due to an obstacle in the field of view, the NIR cameras would not “see” all areas what the TV camera is able to see. Thus Alternative 2 was chosen even though the optical coating of the standard TV lenses is optimized for visible and causes veiling glare (flare) to appear in NIR.

Optics:

The image size and lens mount for the optics in 4K TV cameras to be used in sports were studied. It came out that B4 will be the standard lens mount in sports broadcasting due to larger depth-of-field (DOF).
The initial assumption of the optics pre-study was that high dynamic range (HDR) NIR cameras would be developed. Based on this, the NIR relay lenses with their irises of current implementation can be eliminated such that the HDR NIR camera will always be run iris fully open, resulting in higher NIR intensity on the cameras and simplified field operations but in reduced the depth of field (DOF). Based on our experience we however assume that smaller DOF will not be a problem.

One problem of the current implementation is that the focus plane of the NIR images changes whenever the TV lens focus or zoom are adjusted, bringing the NIR images out of focus. That’s why a very special automatic focusing concept for the NIR cameras was designed (patent applied for).
It was decided to employ regular direct current (DC) motors to actuate the NIR focusing system. The pre-study of optics was carried out together with Carl Zeiss GmbH, Jena, which is also the developer and manufacturer of the current optics.

It came out in the optical pre-study with Zeiss that Zeiss is able to incorporate the optical section of the HDR camera to be a part of the beam splitter, such that in addition of the optical section mentioned, two standard non-HDR cameras per NIR band would be needed. This was a major achievement leading to lower cost and smaller size.

An investigation of image sensor options was made. It was decided to choose the NIR enhanced Onsemi Python5000 sensor and Sony IMX255 for further testing, although the latter one was not in production then. The existing close “relative” Sony IMX250 with the same pixel construction and size but with smaller number of pixels could however be used in testing.

In order to achieve high sensitivity, initially the cooling of NIR image sensors seemed to be absolutely necessary. This was because the dark noise of the image sensors doubles for every 7 degrees centigrade increase in temperature, and high outdoor temperatures are expected in some hot countries when the camera system is subjected to direct sunshine. This would be a major technical challenge since Peltier elements used for heat transfer have poor efficiency.

The testing however pointed out that the dark noise of the IMX250 was so low that cooling it down to room temperature or even lower would not provide any benefit. Moreover, the IMX250 sustained the sharpness in NIR, whereas Python5000’s image got smeared to some degree in NIR. The testing results favor the IMX255 over Python5000 even though the latter has slightly better SNR.

Since Periodic Report 1 the technical requirement specification has been completed, the solutions for technical challenges, such as suspension of the moving optical blocks, efficient cooling of the housing, appropriate splitting the near infrared (NIR) beams in low and high intensity beams to achieve high dynamic range have been found. The implementation of software for image processing and for various control activities is proceeding.

Overview of project results:

The architecture based on separate NIR lens(es) was rejected because it was found to be impractical. The architecture resembling the current one was chosen as it is a proven concept. The new HDR NIR cameras will have a smart construction and high sensitivity. Yet they can be focused independently of visible focusing. The space requirement will be much less than with the existing solution. Due to smarter size and operation and improved performance, some new application areas of augmented reality may open up. The application in sports will become much easier.

Overview of the progress:

The project is in the schedule that was revised and accepted in October 2016. The WP1 along with technical requirement specification was completed. The implementation is proceeding smoothly. The third generation mechanical prototype will be completed during April, and the first fully working prototypes will be completed in August 2017.

The commercial contacts:

The National Hockey League invited Supponor to participate in World Cup in Toronto, in November 2016. The second generation mechanical prototype was also shown there. The participation in the broadcasting was a very considerable success to Supponor and convinced the interested parties in Supponor’s ability to deliver good quality and resulted in many new contacts. Many discussions about co-operation with several parties are now going on. The activities mentioned create a very good background for presentation of the system under construction.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The new HDR NIR cameras will have a smart construction and high sensitivity. Yet they can be focused independently of visible focusing. The space requirement will be much less than with the existing solution. Due to smarter size and operation and improved performance, some new application areas of augmented reality may open up. The application in sports will become much easier.

Related information

Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top