Lights, cameras, action for advanced 3D visualisation
3D visualisation enables a wide range of specialists, from artists to scientists, to represent and virtually manipulate their work. Yet challenges to reaching highly-realistic displays remain. Stereoscopic displays (using glasses) presenting two 2D images (with left and right horizontal disparities), leave viewers still seeing the same scene when they move. Autostereoscopic displays, while allowing users to dispense with glasses and generating some scene changes; on movement reveal the borders between image pairs (‘banding artefacts’). Increasing visual density with light-field projections addresses this problem, but at the cost of the very high data volume required to recreate anything realistic-looking. The EU funded PROLIGHT-IAPP project tackled this problem of efficiently capturing, analysing, modelling, compressing and rendering real-world scenes. It did so by exploiting modern signal processing methods, complemented by an understanding of light and light field properties, recreating a large scale ultra-realistic visual scene. Achieving full parallax real-world 3D visualisation To perceive the world in 3D a number of visual cues are employed. As the project coordinator Professor Atanas Gotchev explains, ‘Firstly, both eyes see slightly different perspectives with retinal disparities processed by the brain to make sense of scene depth. Secondly, both eyes focus on the object of interest and the feedback of how the lenses focal length changes, forms the cue. When people move, they change the view perspective and the relative motion of viewed objects in relation to the background, depends on their distance. There are also other cues such as perspectives, shadows, relative sizes, etc. which can be seen by a single eye.’ With these cues working together, ‘full parallax’ is achieved, where viewers perceive simultaneously the correct perspective, correct focus and motion parallax when they move horizontally and vertically. Light field displays are able to achieve it through a dense enough set of light rays, hitting the retinas in the same way rays from real objects do. To be able to generate such a dense set of rays, PROLIGHT-IAPP had to first capture information about light ray location and direction, and to generate scene structure and geometry. It did so by experimenting with combinations of capture settings from various sensing devices. These included matrix camera rigs capturing horizontal and vertical parallax change; combinations of cameras and Time-of-flight (ToF) sensors (measuring camera to objects distances), creating texture and depth maps; as well as coded apertures using masks to glean scene information based on focus/defocus knowledge. As Prof Gotchev summarises, ‘We demonstrated that we can fully reconstruct the continuous light field from sparse camera setups and render all the needed rays using some novel computational imaging and light field approximation approaches’. The team also sought to quantify the head-motion parallax and discovered that when test subjects were shown sinusoidal patterns varying in depth, in stereo and mono, people distinguished depth changes twice as effectively when motion parallax (seeing different perspectives when moving the head) was present. Benefiting European citizens in the digital era For now, despite having the know-how, the next-generation of ultra-realistic displays are out of reach due to the price of components such as high frame rate projectors. However, in preparation PROLIGHT-IAPP has developed some of the necessary hardware, as well as a new display calibration method - important bearing in mind the 70 million rays in a current light field display and the 5 billion or so rays needed for full parallax. Explaining the driving forces for this new digital technology, Prof Gotchev says, ‘Light field imaging is the next big thing in audiovisual media and has the potential to change the way people consume and interact with visual information.’ Yet in line with the EU’s, ‘Digital Agenda for Europe’, seeking a single market for the digital exchange of content and services, these visual technologies also benefit scientific research and applications, with physicians for example being able to represent anatomic parts for improved prevention and treatment of diseases, as well as for education. The professor goes on to extrapolate that, ‘Light field imaging is also of vital importance in areas where human presence is risky or impossible, for example dealing with natural disasters. Additionally, it is of use for visualising data otherwise considered quite abstract, such as Big Data.’
Keywords
PROLIGHT-IAPP, 3D visualisation, light-field projections, light properties, full parallax, motion parallax, display bandwidth, digital technologies, ultra-realistic visualisation
