Community Research and Development Information Service - CORDIS

H2020

DigiArt Report Summary

Project ID: 665066
Funded under: H2020-EU.3.6.3.

Periodic Reporting for period 1 - DigiArt (The Internet Of Historical Things And Building New 3D Cultural Worlds)

Reporting period: 2015-06-01 to 2016-05-31

Summary of the context and overall objectives of the project

DigiArt’s principal aim is to make the large scale 3D capture of cultural assets and their subsequent presentation in an augmented or virtual reality setting a practical reality. It seeks to achieve this by making significant advances in the flowing areas:
1. The efficient 3D capture of cultural heritage artefacts of widely varying scale ranging from desk top to macro-geographic sites. Also ranging from the accessible to the less accessible.
2. Providing means for the effective manipulation of such 3D data including in particular the registration and combination of 3D data originating at different times and from different modalities.
3. Improving the efficiency of curating digital collection via the development of semi-automated tools for the semantic classification of 3D digitised objects.
4. The development and demonstration of tools for the creation of narratives that will unfold in AR (augmented reality) or VR (virtual reality), allowing the interlinking of captured artefacts, perhaps physically separated in reality, in informative, educational and illustrative scenarios.
In addition to these technical aims DigiArt has workpackages and activities designed to ensure active user group engagement, effective dissemination and appropriate exploitation.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We would assess progress in all areas of the project in the first 12 months to have been excellent with all deliverables due in the period being achieved and indeed achievements in many areas being beyond that projected and expected at this point.
Particular highlights of the first year’s activities include:
1. The full 3D capture of the prehistoric cave at Scladina, Belgium and the subsequent development of three 3D models of the site at various levels of resolution. This scan was we believe the first ever of a cave (a GPS denied environment) to be carried out using UAVs (unmanned aerial vehicles) alongside more traditional ground based photogrammetry. It also included multimodal 3D capture with SfM (shape-from-motion) photogrammetry being combined with LIDAR scanning. The data capture of this site, at high resolution, was carried out in just two days – much faster than any previous comparable scanning task we are aware of.
2. The 3D capture of the UNESCO World Heritage Site of the Palace and Royal Tombs of Philip II of Macedonia at Aigai in Greece. This 1.65Km2 site was fully captured by UAV photogrammetric scanning and LIDAR in just 4 days.
3. The project has successfully developed a low cost, open source, desk top scanner. At present this is at prototype-test level but is already successfully 3D scanning various smaller artefacts. Acquisition times are less that 10secs with full processing of a 4M point (x,y,z) measurement field in under 30 seconds. This is achieved with no human intervention in the process at all.
4. The first year of the project has successfully achieved the synthesis of SfM data with LIDAR data from the cave. More remains to be done in this area but this is notable early success within DigiArt’s multi-modal data registration theme.
5. Considerable work has been done to develop exciting and interesting “story boards” to be placed within the virtual worlds created by the project’s innovative story engine. These story scenarios centre around Scladina and Aigai and have been developed by the specialist staff at those centres in dialogue through the users group with the technical teams in DigiArt. These are the foundations for the first generation VR/AR demonstrators which will be achieved in the second year of the project.
6. As well as the close engagement of the two demonstrator sites the project has worked hard to reach out to a wide spectrum of different types of museum to give them some experience of DigiArt technology. In this first year four such outreach museums have been engaged with and these include; two archaeological sites, one historic park and a large scale industrial heritage artefact. These represent a wide range of challenges to DigiArt’s scanning technology and all of them have been successfully 3D scanned.
In addition to these specific activities the project has engaged in a wide range of dissemination and promotional activities. These include; academic technical publications, media events, public exhibitions, the development and release of eleven demonstration videos and the presentation of the project at various events.
The management of the project in the first year has been effective and ensured smooth operation. There have been four Consortium and Project Management Team meetings in the course of the year, in Liverpool (June 2015), Belgium (Sept 2015), Switzerland (Dec 2015) and Greece (May 2016). All partners have conducted their own financial management and all have returned the appropriate reports.
The project is fully on schedule and no major problems or issues have occurred in the first year.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Although only having just completed its first year DigiArt is already beginning to show signs of progress beyond the state-of-the-art in some areas. These are briefly outlined below:

Comparative UAV and Ground-based 3D Scanning.
Ground-based scanning, achieved by mounting cameras in an array of positions mounted on tripods and taking a series of images, is a well-established technology. As such it acts as baseline standard. Airborne 3D scanning using a derivative of photogrammetry – i.e. shape from motion – is by comparison much more recent. Very little has been done to compare results from these two technologies. DigiArt is starting to gather valuable experience and data in this area. The Scladina Cave was fully scanned using both techniques – this provides a unique data for comparison purposes. The Daniel Adamson Outreach project is partially UAV scanned and partially ground-based scanned, with overlapping areas. This not only provides a basis for comparison, but it also offers experience in combining data originating from the two methodologies. To do this successfully we must fully understand the error distribution statistics, differing datum points and contrasting natures of the two data sets. Data from Scladina also assists in this task.

Combined Scanning Modalities.
DigiArt is starting to get systematic experience by gathering data in the field of multimodal scanning. At Scladina and at Aigai we have data from both photogrammetry and LIDAR. We are beginning to develop the capability to merge these two modalities. This is not trivial problem, but solving it offers great potential. Airborne SfM is very fast, it can capture large areas in a semi-autonomous fashion in a fraction of the time of alternative techniques. But it is challenged by fine detail. LIDAR is excellent at fine detail, but it finds large areas difficult to cope with, often producing far too much data which differs little and so has a poor information density. Combining these modalities offers us “the best of both worlds”. As a result of work done during the first year of the project LJMU-GERI have just completed the design and build of a combined SfM/LIDAR sensor which can be lifted by a UAV. This is the first of its type anywhere and will be tested during year two of the project. The accompanying picture below shows the new combined sensor.

New Flight Path Design methodologies to Optimise SfM.
The airborne scan of the cave using UAVs yielded some excellent data. The resultant 3D model is of very high quality. But the processing time required to produce it was excessive. We could not in all seriousness accept that as a regular overhead. Investigations were carried out to determine why the processing was taking so much time. This revealed that the principal problem was not the large number of mathematical operations required, but the constant swopping in and out of memory of images in the sequence separated by considerable temporal distance and only small physical distances. In order words the images in the video stream were in a severely sub-optimum arrangement. This occurred because the flight path followed by the UAV did not seek to optimise the image sequence. We have been working on techniques to do this. Early indications form the Aigai data would suggest that we have been successful. If so, we have new flight planning algorithms for UAVs engaged in SfM which would have wide application in areas such as; mapping, surveying, surveillance, infra-structure monitoring etc.

GPS Denied Enclosed Space 3D Scanning.
We have shown, for the first time we believe, via our UAV scan of the Scladina cave, that it is possible to gather comprehensive 3D data on the boundaries of a completely enclosed space which is GPS denied. This has implications outside of cultural heritage applications.

The Desk-top Open Source Scanner.
The realisation of the new desk-top, static, scanner with its Open Source software is a first. It offers the prospect of ultra-low-cost 3D scanning capability for every museum and gallery in Europe. The device is simple to construct, the software is robust, the measurement cycle is fast, the data is high resolution and accurate. Developments by the team in new phase unwrapping algorithms (included in the dissemination publications as technical dissemination) have further enhanced its robustness. The next six months will see further facilities added and new algorithms incorporated in readiness for the launch to the user group in month 18.

Using Deep Learning Architectures for 3D Data Analysis.
Recent advancements in 3D sensing technology (e.g. LIDAR and UAV sensors) and the appearance of low-cost devices such as Microsoft Kinect have made the collection of 3D data more feasible and affordable than ever. Based on the scanning device employed for capturing the 3D scene or object of interest, raw data are collected in different forms. UAV scanners get range images from different camera viewpoints. Then, these images are typically combined through a registration process or Structure-from-Motion techniques (SfM) in order to discard noisy data, establish correspondences between them and ultimately generate a unified 3D point cloud for further processing. The increasing abundance of 3D data encouraged the research community to exploit this richer content for addressing several computer vision problems related to understanding 3D scenes, e.g. 3D Object Classification, 3D Object Recognition and 3D Shape Retrieval. Indeed, the possibility of using the additionally provided attributes of depth and full 3D geometry represents an important advantage which can significantly boost the performance of several applications. Until recently, the standard approaches to many vision tasks involved the extraction of hand-crafted local or global descriptors from the available images or videos. The breakthrough results reported in the 2012 ILSVRC image classification task though, changed completely the landscape of computer vision. Neural networks and deep learning now dominate on almost every recognition and detection task making every group of computer vision researchers and practitioners to re-design their systems. Following this trend, one of our main line of activities in the context of ProTail is to use the huge volume of data generated within the project to train deep learning architectures for the tasks of 3D recognition and retrieval. Our survey paper submitted to ACM Computing Surveys constitutes a solid basis for our future work, which is now directed towards replicating the state-of-the-art results and introducing novel methods.

Story Telling Engine in a Flexible Architecture.
In designing the architecture of our story-telling engine, we had to address a number of requirements including expandability, generalizability, user-friendliness, immersiveness, compatibility to several platforms, and the ability for the final product to be commercialised. In the past several attempts to make VR/AR tours were discontinued due to the lack of an appropriate architecture that will be self-sustained and commercially active. The architecture that has been proposed as part of WP6 activities focuses on reducing, as much as possible, the interference of a programmer or a game designer so that the main target group, i.e. the curators, can easily make their own VR/AR tours and maintain it. This was achieved by proposing an architecture that interconnects a web-based interface for 3D content management with the powerful game making engine of Unity. In this way, we were able to abstract the process of setting the 3D scene and defining triggers on the 3D objects, without sacrificing the quality of the resulting game.

Related information

Record Number: 191339 / Last updated on: 2016-11-15