Periodic Reporting for period 3 - ARIESS (Augmented Reality and Indoor Navigation for Enhanced ASSembly)
Période du rapport: 2020-09-01 au 2021-10-31
Today, most of the test operations at the Final Assembly Line (FAL) involve the use of paper and manual operations, like work orders, test results, etc. In the case of troubleshooting tasks or in cases where expert assistance is required, people must be physically at the workplace to collaborate efficiently. The use of Augmented Reality and IIoT is being used more and more in other industries to tackle the problems aforementioned in a more sophisticated way. The introduction of AR in the aeronautic sector is challenging, but it’s becoming more a more real, for example in the manufacture of cable harnesses. Project ARIESS goes a step further by introducing AR in the fuselage of the aircraft, where mobility is a tight restriction. Another challenge tackled by ARIESS is the development of situational awareness in a complex workplace like the FAL. The use of technologies like IPS contributes to improving orientation in the workplace and makes it easier for the workmen to move around. Deploying a secure, reliable and manageable network is the third challenge of ARIESS.
The main conclusions obtained for the three technologies envisioned by Project ARIESS are the following:
-Augmented reality (AR). It was verified that all functionalities and requisites established for the hands-free HMI, based on augmented reality glasses (Hololens), developed in ARIESS project are correct. The use case studied was the full procedure of deploying a test from CATS to the ARIESS GUI running on HoloLens hardware.
-Indoor positioning system (IPS). A new algorithm was proposed to improve the estimation of the position and it has been demonstrated to be a more accurate algorithm than LM in terms of the error position, in real tests. Magnetoresistive sensors have been considered to design the sensor module of the IPS for the ARIESS Project. It was demonstrated that this kind of sensors can be used to estimate the position.
-Industrial Internet of Thing. A new communication architecture and middleware that integrate in an effective way the information distributed in the factory have been development. ThingWorx platform has been used and the IoT software for Raspberry Pi was developed using the tools provided by PTC.
The main conclusions obtained for the three technologies envisioned by Project ARIESS are the following:
-Augmented reality (AR). It was verified that all functionalities and requisites established for the hands-free HMI, based on augmented reality glasses (Hololens), developed in ARIESS project are correct. The use case studied was the full procedure of deploying a test from CATS to the ARIESS GUI running on HoloLens hardware.
-Indoor positioning system (IPS). A new algorithm was proposed to improve the estimation of the position and it has been demonstrated to be a more accurate algorithm than LM in terms of the error position, in real tests. Magnetoresistive sensors have been considered to design the sensor module of the IPS for the ARIESS Project. It was demonstrated that this kind of sensors can be used to estimate the position.
-Industrial Internet of Thing. A new communication architecture and middleware that integrate in an effective way the information distributed in the factory have been development. ThingWorx platform has been used and the IoT software for Raspberry Pi was developed using the tools provided by PTC.
Initially, the states of the art and the definition of the use cases for these WPs were generated. Later, requirements were defined and confirmed by the Topic Manager and the solutions for each Work Package were designed. For WP1 this entailed the study of various possible Augmented Reality technologies, centred around the use of Microsoft HoloLens headset. Also, investigation and development of necessary improvements to the Indoor Positioning System (IPS) presented were performed, focused on improving its performance in the 3D space and when tracking moving objects. For WP2, the design of the hardware elements needed to satisfy the communication requirements was performed with a combination of off-the-shelf and custom designed elements. Communications with the Topic Manager’s server infrastructure where also defined, as well as integration with the IPS system.
Later, the solutions were implemented for both Work Packages.
For WP1, the focus was the development of an innovative Augmented Reality HMI. The HMI is based mainly on Microsoft HoloLens, the Unity engine and the MRTK (Mixed Reality Tool Kit) plugin. A first prototype version of the HMI was developed and functionalities such as user interaction, voice recognition and positioning and guidance assistance were validated successfully. Moreover, a Middleware Network Protocol (MNP), designed to work over TCP/IP, for communication was defined and implemented as a standalone application first. Once the first prototypes were validated, the full implementation and integration of this technologies were performed. For the HMI, a full upgrade of its design was performed, and new capabilities were included. For MNP, both the integration of the protocol with the CATS test system and with SKYLIFE’s own HMI were developed. Due to lack of compatibility between the standard socket libraries used on HoloLens and CATS (standard C#), an additional development was necessary, using one of the multiplayer communication protocols from Unity (UNET) to serve as a bridge between the CATS application and HoloLens. Finally, in the validation phase, work was done to provide an adequate testing environment at Skylife’s facilities, satisfying the requirements that would be presented by the real FAL as much as possible. For this final validation tests, the test scenario was improved with the inclusion of an Augmented Reality model of a real aircraft cockpit and project was upgraded to also work on the more recent HoloLens 2 hardware, to prevent obsolescence and guarantee the impacts of the project.
For WP 2, the full development of a custom hardware capable of IoT communication (based on ThingWorx), communication with a number of existing tests means and the ability to position itself on an industrial environment was performed. Regarding infrastructure development, the SILEME Indoor Positioning System was adapted to work on the COTS hardware selected and its position estimation was improved with the development of new filters. Moreover, IoT communication via ThingWorx was implemented and tested with the data coming from the IPS itself. Hardware and Software development of the ARIESS HUB system developed. This includes the assembly of a prototype unit based on the Raspberry Pi Zero platform, the integration of all previous developments. The implementation of the IoT/Thingworx communication was completed and fully specified. To improve usability, the ARIESS HUB was equipped with an OLED screen to display information to the user such as error codes or data gathered. Finally, in the validation phase, tests were conducted at Skylife’s facilities, against a testing ThingWorx server deployed for the occasion. Tests were performed focusing in communication with test sensors, battery life, IPS precision, communication with ThingWorx, etc. As an additional work on this phase, a 3D printed enclosure for the ARIESS HUB has been developed, to serve as the basis for a future production run.
Later, the solutions were implemented for both Work Packages.
For WP1, the focus was the development of an innovative Augmented Reality HMI. The HMI is based mainly on Microsoft HoloLens, the Unity engine and the MRTK (Mixed Reality Tool Kit) plugin. A first prototype version of the HMI was developed and functionalities such as user interaction, voice recognition and positioning and guidance assistance were validated successfully. Moreover, a Middleware Network Protocol (MNP), designed to work over TCP/IP, for communication was defined and implemented as a standalone application first. Once the first prototypes were validated, the full implementation and integration of this technologies were performed. For the HMI, a full upgrade of its design was performed, and new capabilities were included. For MNP, both the integration of the protocol with the CATS test system and with SKYLIFE’s own HMI were developed. Due to lack of compatibility between the standard socket libraries used on HoloLens and CATS (standard C#), an additional development was necessary, using one of the multiplayer communication protocols from Unity (UNET) to serve as a bridge between the CATS application and HoloLens. Finally, in the validation phase, work was done to provide an adequate testing environment at Skylife’s facilities, satisfying the requirements that would be presented by the real FAL as much as possible. For this final validation tests, the test scenario was improved with the inclusion of an Augmented Reality model of a real aircraft cockpit and project was upgraded to also work on the more recent HoloLens 2 hardware, to prevent obsolescence and guarantee the impacts of the project.
For WP 2, the full development of a custom hardware capable of IoT communication (based on ThingWorx), communication with a number of existing tests means and the ability to position itself on an industrial environment was performed. Regarding infrastructure development, the SILEME Indoor Positioning System was adapted to work on the COTS hardware selected and its position estimation was improved with the development of new filters. Moreover, IoT communication via ThingWorx was implemented and tested with the data coming from the IPS itself. Hardware and Software development of the ARIESS HUB system developed. This includes the assembly of a prototype unit based on the Raspberry Pi Zero platform, the integration of all previous developments. The implementation of the IoT/Thingworx communication was completed and fully specified. To improve usability, the ARIESS HUB was equipped with an OLED screen to display information to the user such as error codes or data gathered. Finally, in the validation phase, tests were conducted at Skylife’s facilities, against a testing ThingWorx server deployed for the occasion. Tests were performed focusing in communication with test sensors, battery life, IPS precision, communication with ThingWorx, etc. As an additional work on this phase, a 3D printed enclosure for the ARIESS HUB has been developed, to serve as the basis for a future production run.
The expected results and potential impacts are those in the original proposal, for the current stage of the Project.
ARIESS introduces a number of innovations in the Final Assembly Line of an aircraft that makes a qualitative difference with respect to the current processes and methodologies. The
solution proposed is a step forward in the degree of technification of the modern FAL, in areas with a lot of room for improvements like functional tests of fixtures or parts in the aircraft and MRO operations.
ARIESS will be a leap forward in functional tests and other tasks in the FAL with regard to lead time, recurring costs and also flexibility and transparency. The main contribution of the technologies addressed in ARIESS is to contribute and speed up the introduction of automation in aircraft manufacturing and thus to the identified challenges: Reduction of Production Cost, Improved Quality and Reliability, Better Floor Space Utilization, Reduce Waste, Stay Competitive.
Given that final test of the solutions obtained in ARIESS couldn’t be tested at the real worksite due to various reasons, the expected immediate impacts are affected, as further tests will be necessary afterwards to bring these technologies to the level of readiness desired.
ARIESS introduces a number of innovations in the Final Assembly Line of an aircraft that makes a qualitative difference with respect to the current processes and methodologies. The
solution proposed is a step forward in the degree of technification of the modern FAL, in areas with a lot of room for improvements like functional tests of fixtures or parts in the aircraft and MRO operations.
ARIESS will be a leap forward in functional tests and other tasks in the FAL with regard to lead time, recurring costs and also flexibility and transparency. The main contribution of the technologies addressed in ARIESS is to contribute and speed up the introduction of automation in aircraft manufacturing and thus to the identified challenges: Reduction of Production Cost, Improved Quality and Reliability, Better Floor Space Utilization, Reduce Waste, Stay Competitive.
Given that final test of the solutions obtained in ARIESS couldn’t be tested at the real worksite due to various reasons, the expected immediate impacts are affected, as further tests will be necessary afterwards to bring these technologies to the level of readiness desired.