Scale-up of INNER MAGIC and INNER REFLECTION: Virtual Ultrasound Sensors for Smart Devices.

Project Information

IMIR-UP

Grant agreement ID: 849692

DOI

10.3030/849692

Project closed

EC signature date 30 January 2019

Start date 1 February 2019

End date 31 May 2020

Funded under

• INDUSTRIAL LEADERSHIP - Innovation In SMEs

Total cost

€ 3 087 495,00

EU contribution

€ 2 161 246,50

Coordinated by

ELLIPTIC LABORATORIES ASA
Norway

Periodic Reporting for period 2 - IMIR-UP (Scale-up of INNER MAGIC and INNER REFLECTION: Virtual Ultrasound Sensors for Smart Devices.)

Reporting period: 2019-10-01 to 2020-05-31

Summary of the context and overall objectives of the project

As smart devices and home appliances have become increasingly popular, the device manufacturers require constant innovation in order to improve their products while simultaneously reducing costs. Functionalities such as preparing for interaction when use is sensed present, and intuitive controls are often desired but no easily solved. Hardware-based solutions are expensive, requiring dedicated sensors and long deployment times to fully integrate them into devices, since they must be manually tuned and calibrated for each new type of application. This makes it difficult to customise such devices for different market segments and end-user needs. Furthermore, extra hardware requirements often compromise a device’s design, increasing their bulk and rendering them old-fashioned. Many hardware-based solutions also increase the risk to a user’s privacy and data security, such as those solutions involving cameras. As a result, manufacturers are adopting software-only solutions instead of hardware. However, they do not yet have a presence detection or gesture recognition solution ready for mass commercialisation. The IMIR-UP project will provide the right solution to this challenge.

Elliptic Labs is the world leader in ultrasound sensing software for smart devices, enabling touchless interactions and gesture interpretation. INNER MAGIC and INNER REFLECTION (IMIR) products are the first-of-their-kind integrated solutions for non-intrusive user presence detection and intuitive gesture recognition, transforming smart devices into genius devices. The aim of the IMIR-UP Phase 2 project is to fully mature our award-winning and patented technology for easy, mass-scale (“plug-n-play”) deployment in any consumer electronic smart device without requiring supplemental hardware (software only). Bringing our innovative IMIR products to the mass market will positively disrupt existing touchless sensing technologies, resulting in an improved user ability to intuitively interact with their smart devices via a natural extension of normal human behaviour.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The main focus of the first half of the project had been on maturation of the core technology and building and expanding the client and channel partner base for scale-up.
Components that constitute the overall cloud-based ML (machine learning) platform for IMIR-UP had been developed alongside and tested through the process of developing prototypes and demonstrators requested by the customers. This ensures the use cases are defined from real world problems and with specific requirements in solutions. In the second half of the project, performance, usability, efficiency and cost benefits of the ML platform had been extensively tested and evaluated. The prototypes and demonstrators had been validated by the customers and confirmed their superior and robust performance.
At project end, the ML platform contains the ML training and build pipeline (including signal processing modules), Web-based tools, backend infrastructure and frontend user interface. The costumers have the possibility of using plug-n-play functionality to evaluate the software on processing platforms and acoustic configurations supported by the platform.
In addition to direct sales, participation at main industry events had presented results to various customers and key stakeholders.
Currently presence sensing and touchless gestures for laptops and smart devices are the main market segments.
Extensive press releases had highlighted the formal partnerships formed with key industry players such as MediaTek, Qualcomm, Infineon and ARM. These partners have been particularly important for commercialisation of IMIR software products. To promote our presence at the local research and scientific communities, open house events had been organised, along with talent acquisition through presentations at university career days.
Continuous effort has been put into developing intellectual property and expanding the patent portfolio to ensure our freedom-to-operate.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The state-of-the-art competing technologies in the domain of presence sensing and touchless gestures include vision based systems using cameras, time-of-flight (ToF) sensors using infrared (IR), radio frequency systems such as radar and WiFi, and touch screens. Without exception, these technologies require dedicated hardware sensors, and additional processing software, which means there is substantial cost and challenges in integration. Certain technologies such as camera, ToF sensors and touchscreens have various limitation in range and coverage. Cameras in addition lead to user privacy concerns.
There has been significant progress made compared with the competing technologies through the project. The key points are summarised as follows:
a. developed fully embedded, integrated ultrasound-based virtual sensors for presence sensing and touchless gestures which do not require additional hardware and instead use only built-in speaker and microphone
b. validated robust performance achieved using machine learning
c. building cloud-based ML platform which enables easy scale-up
d. supporting sensor fusion which incorporates other sensors to further improve performance and provide additional functionalities.

At the end of the project, the cloud-based ML platform will be ready to launch. Through specifically designed interface, customers will be able to provide use case definitions, product and performance specifications, all can be done remotely. Elliptic Labs then provides a software integration package specific to their devices, in addition to a set of machine learning classifiers, which are built using the collected training data sets and various signal processing modules. With the set of web tools provided, customers also have the possibility to gauge and optimise performance by collecting data themselves. Through out the process, Elliptic engineers will provide continuous customer support for specific integration and optimisation.

Because the platform is cloud-based, and is built on top of Elliptic's ever expanding knowledge and data base, it can benefit the customer by greatly reducing the deployment time and cost, while at the same time allows Elliptic to support its ever growing customer base with superior products. The expected economic impact includes entry in the global billion dollar IoT market, gross profit, generating employment and stimulation of the R&D activities in machine learning and ultrasound technology.
In addition, waste of electrical and electronic equipment (WEEE) is one of the fastest growing waste streams in the EU. It grows at 3-5 % per year and is expected to account for more than 12 million tonnes by 2020. In 2012 the EU approved the WEEE Directive to reduce the quantity of such waste to be disposed of. The IMIR-UP Phase 2 project will contribute to the WEEE Directive as a software-only solution that does not need supplemental hardware, which is not the case of the other alternative technologies. Furthermore, the EU has set different directives to improve energy efficiency in electronic products (e.g. the EU Directive 92/75/EC that established an energy consumption labelling scheme) and lighting systems (e.g. EU Directive 874/2012 that established energy labelling regulation for lamps and luminaires). INNER REFLECTION will contribute to improve energy efficiency of electronic products and lighting systems.

Benefits of IMIR-UP