AuThoring tooL for indoor Augmented and dimiNished realiTy experIenceS

Project Information

ATLANTIS

Grant agreement ID: 951900

DOI

10.3030/951900

Project closed

EC signature date 20 May 2020

Start date 1 July 2020

End date 30 June 2022

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 1 756 768,75

EU contribution

€ 1 507 200,63

Coordinated by

JOANNEUM RESEARCH FORSCHUNGSGESELLSCHAFT MBH
Austria

Periodic Reporting for period 2 - ATLANTIS (AuThoring tooL for indoor Augmented and dimiNished realiTy experIenceS)

Reporting period: 2021-07-01 to 2022-06-30

Summary of the context and overall objectives of the project

In many application areas (e.g. interior design, furniture retailing or renovation), communication with a customer or future user during the planning and design phase is crucial to select the right products and configurations. Making this communication process effective saves costs, avoids later modifications, and results in providing tailored solutions and higher customer satisfaction. Experts usually express their ideas in traditional 2D drawings produced by computer aided design (CAD) software, making it difficult for the consumers to comprehend them. Emerging technologies such as augmented reality (AR) have the potential to make this process more effective, and put consumers in a better position to review options by experts, or express their ideas, directly in the target environment.

While a number of AR indoor planning apps already exist, there are two main issues that are not yet satisfactorily addressed: First, creation of the room layout needs to be done manually in most apps, or via importing a CAD model. This requires some IT and technical skills, and may be an obstacle for users. Second, indoor design does in many cases not start from scratch in an empty room, but makes changes to an existing room. When viewing changes on site, the realism of AR is severely degraded when the overlaid 3D objects added to the scene clash with real objects.

ATLANTIS addresses the first issue by simplifying the capture process, requiring only a single panoramic image. Extracting semantic information about the room from this image is enabled by automation. Recent advances in artificial intelligence (AI)-based visual scene understanding enable this automation for constrained environments (such as private or office indoor scenes). ATLANTIS addresses the second issue by diminished reality (DR) technologies, which enable visually concealing real objects, a functionality not yet widely found in AR apps.

ATLANTIS mainly targets the interior design sector, albeit its core technologies can be easily employed in vicinity sectors, including applications in tourism, renovation, real-estate, museums etc.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The system design (defined in WP2) follows a layered approach, clearly separating the frontend application, data handling and backend processing services. The user facing application is implemented as a mobile app. Based on the findings from requirements analysis there is no separation between authoring and AR/VR viewing, but all functionalities are integrated, and the user can quickly switch between floor plan, 3D and AR/VR views.

The app includes functionalities for setting up and editing planning projects, to capture the room from 360° images, to insert furniture objects from catalogues and to modify/configure them. The backend is designed as a distributed, service-oriented and event-driven system, serving a mobile user application, and where processing is event-based, triggered by user interactions or availability/modification of data.

Most AI and processing functionalities are provided as services, each of them performing one clearly specified function on a single or multiple data item(s). Layout estimation takes a panorama as input and provides metadata describing the room layout (sparse/planar scene geometry). Instance segmentation takes a view or a panorama as input and provides metadata with object bounding polygons and class labels. Depth estimation takes a single panorama as input and provides an estimated depth map (dense scene geometry) and the reconstructed 3D model of the scene involved. Panorama inpainting takes a view or a panorama, as well as object masks as input and provides a set of image patches or a newly composited panorama for replacing each of the objects. In addition, it provides the dense scene geometry for the non-furnished version of the input panorama. This service implements the DR functionality maintaining the reality and geometry of the scene. Scene localisation uses a panoramic image and the instance segmentation and depth estimation results to propose a set of image patches in the scene which are suitable as image anchors, which the AR app uses for registering the AR scene. Scene graph estimation generates a parametric description of the furniture items in the room and their relative positions, based on results from instance segmentation. This information can be fed into the layout proposal service to complement furnishing of a room or suggest alternative layouts.

Throughout the project, prototypes have been tested with representatives of relevant user groups. Two larger rounds of evaluations have been performed, comparing the tool developed in ATLANTIS with two state of the art baselines, and proving the validity of the approach taken by the project. In addition, guided walkthroughs with professionals from different backgrounds have been performed to gather wider feedback.

Towards exploitation, the project has developed a business plan, set up a stakeholder group with members from target user domains, and has presented results at online and offline events.

In addition to public deliverables, brief reports summarizing results have been published at https://atlantis-ar.eu/result.

Datasets and green open access copies of publications are provided at https://zenodo.org/communities/atlantis-ar/ and many results of the project are available as open source software (e.g. at https://github.com/atlantis-ar/).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project has advanced the state of the art in algorithmic developments for scene understanding and inpainting. The user tests have shown that the solution provided by ATLANTIS outperforms existing tools in terms of efficiency and user satisfaction.

ATLANTIS enables a more cost-effective creation of content for AR, requiring only basic IT skills. Following the project, the solution will be deployed in a specific application domain (interior design and furniture sales), but this is a domain involving a broad range of companies and every citizen as a potential consumer. Thus the number of consumers gaining experience with XR technologies will increase. The experiences gained can serve as a blueprint for adapting the authoring approach to other domains sharing some of the requirements. This includes industrial production planning, automotive industries but also the cultural heritage sector.

In addition to the SMEs involved in the consortium, the technologies developed in ATLANTIS will benefit a large number of companies in the interior design and related retail businesses, which are to a large extent SMEs. Deploying the solution in other sectors will reach further SMEs, and stimulate adapting the solution to fit the specific requirements of these sectors. ATLANTIS will further extend the market opportunities of the two SMEs in the consortium. ROOM reaches a large set of consumers via their large base of B2C customers, and can improve their offer to their current customer domains, as well as reach out to B2B customers in other industries.

The additional features and higher degree of automation in content creation will facilitate takeup of the solution in other application domains. Usability Partners will extend its service offering to include usability studies on XR tools and applications.

Demo room at Roomle, with furniture diminished

Demo room at Roomle

Automatic segmentation of indoor objects in panoramic image

ATLANTIS app: projects

Panorama of test lab, with some objects diminished

ATLANTIS app: diminished object

Estimation of room layout from single panoramic imagge

Panorama of test lab

Estimation of depth map from single panoramic image

ATLANTIS app: floorplan view

ATLANTIS system overview