Final Report Summary - RANGE-IT (Detection and multimodal presentation of indoor objects for visually impaired people)
Executive Summary:
For the Millions of people who are blind or visually impaired, moving around in dynamic indoor environments can be very challenging. Their perception of the world usually ends as far as their white cane reach.
Multiple Electronic Travel Aids (ETA) are available as assistive technologies, which help to avoid obstacles and hazards. Many of these early ETAs use ultrasonic sensors or digital cameras to detect obstacles.
Most of those existing prototypes and commercial products only present whether there are obstacles on their way, however, they fail to provide a comprehensive situation awareness function which let blind and visually impaired people better understand their surrounding environment.
The 3 most important problems the current version of the Range-IT system aims to solve are:
1. Spatial awareness of visually impaired people ends at the end of the white cane
2. Current products do not provide a comprehensive situational awareness to understand their surrounding environment.
3. Audio based systems interfere with the main information channel available to the visually impaired.
The Range-IT system allows blind and visually impaired users to understand the surrounding indoor environment. A 3D time-of-flight camera scans the environment for stairs, obstacles (hanging, standing or moving). The camera is able to detect a range up to 5m which significantly enlarges the spatial awareness of the visually impaired in comparison to a white cane.
The Range-IT system can;
- Detect the existence of obstacles ahead of the user beyond the reach of the cane up to 5 meters when walking
- Detect the existence of obstacles in a virtual corridor left and right.
- Detect hanging obstacles that can cause collisions.
- Support users in finding stairs that lead upwards
- Support users in finding open doors
- Support users in getting a basic picture of the structure of a room
- Support users in getting an idea about the distance to a wall and its orientation.
The user is informed about the different obstacles by means of tactile (vibration) patterns and by interactive spatialized sounds. A big advantage of tactile representation is that the user can hear environmental sounds and obtain information about obstacles simultaneously. Relevant information can be given via small vibrations motors on the belt, without overloading the user. Through the spatialized signification sounds, blind and visually impaired users are able to quickly acquire direction and distance information surrounding obstacles.
Project Context and Objectives:
The RANGE-IT system is a stand-alone wearable prototype for the blind and visually impaired, which detects and presents close range standing, and hanging objects to the user in indoor environments. Furthermore, specific reference points (e.g. stairs, walls and open doors) can be identified. Information is presented in tactile and/or audio modality based on the context.
The market for Assistive Technology (AT) addresses a growing number of European citizens, as demographics change and more and more people grow older. Economics in this domain consists of a mixture of private and public sources and is bound to specific activities including education, work and daily living. For example blind or visually impaired people are often supported by public funding regarding the purchase of technological assistance systems, which shows the public concern to support activities necessary for an independent life, such as reading printed documents through scanners and speech synthesizers. The World Health Organisation states that one of 175 people is totally blind (i.e. visually impaired people are not included). Although there is some assistive technology available for blind people at the work place and at home, this doesn’t include assistance in mobility in other places. The majority of blind people are only mobile if they have help of others (in person or by mobile phone), or use solely known pace ways. Besides the cane with a limited range and an expensive guide dog, there are very few aids that support individual mobility of the blind.
A product aiming at mobility addresses not only people at the workplace but may reach all blind people, if they can afford it or see its cost justified. If compared to the functionality of a colour checker (a device naming colours) the price will be slightly higher but many more benefits will be gained. Moreover, as a ramp technology, it is expected also people with low vision want to feel safer in unknown environments through the RANGE-IT system.
The prototype of the RANGE-IT system is developed using iterative and parallel research determining the multiplicity of user requirements in different contexts and matching it with device and user capabilities. With regard to user capabilities, The RANGE-IT system will for example add only little additional effort to the already high demand on the blind person to observe the environment through hearing sound scenes and touching the ground by feet.
The white cane can detect the ground surface including changes in the ground texture and it emits sound allowing the walker to monitor changes in the environment such as bushes, walls and openings. It cannot detect obstacles above the waist and nothing beyond the tip of the cane. The reachable space (range) of an arm and cane by turning around is about 3.14 m2 (1 m x 1 m x 3.14). RANGE-IT extends this range to 5m, . In the large detectable area, multiple obstacles can be captured and located simultaneously. Thereby, it is possible to acquire the overview layout of surrounded obstacles.
Compared to the conventional strategy of a blind person to "scan" the inside of a huge shopping mall by walking around and performing trial and error movements, the number of necessary trials can be reduced drastically by extending the range. Furthermore, during mobility training, blind people learn how to move along house walls, fences, pedestrian walks, etc. These structures create division lines. Unlike sighted people who perceive division lines visually while moving along them, blind people re-create them from auditory and tactile perception along their own movement. RANGE-IT represents those structures in a virtual and accessible way without the need for the blind person to move all the way along them.
In sum, the objectives of the RANGE-IT system are:
• Extension of range of perception. RANGE-IT has a range of up to 5 meters, which will be a world of difference to the user.
• Empowerment. RANGE-IT will empower visually impaired people to act more independently in society.
• Low cost. The RANGE-IT system will be low cost given the high accuracy. There is no comparable aid on the market.
• Independence. Visually impaired people can act more spontaneously and independently. For example the following activities can be performed more independently: general mobility; visiting authorities; shopping; using public transport.
• Assistance. Less (human) assistance is needed.
• Participation. Mobility is connected to taking part in social life. Increasing mobility thus supports social participation.
Project Results:
1. Specification of the Range-IT system
For the design steps of the system we refer to the confidential deliverables in WP1, and for the detailed results to the confidential deliverables in WP2 to WP5. An overview of the system and of the capabilities of the obstacle detection is given here.
1.1 Overview of the Range-IT System
1. Main Hardware Components:
a. A Softkinetic infrared time-of-flight 3D camera
i. Size (W x H x D): 105 x 30 x 23 mm
ii. Power: < 2.5 w
iii. Illumination type: Diffused laser, For indoor use
iv. Range: 0.15 – 7 m (short range: 0.15 – 1.0m)
v. Resolution: 320 x 240
vi. Frame Rate: 25 – 60 fps
vii. Field of view (H x V x D): 74° x 58° x 87°
b. An ARM-based ODROID-U3 embedded board
i. 1.7GHz Quad-Core CPU
ii. 2GByte RAM
c. An Elitac tactile belt with multiple vibratos
d. A bone-conduction earphone
e. A wired controller with 3 keys
f. A common 5V Li-ion battery with USB output
g. A specific designed harness
2. Power consumption:
a. 5V input
b. Max. 2.9 A
c. System runtime for 6 hours with a full 20Ah 5V Li-ion battery
3. Weight: 1.5kg (including the backpack)
1.2. Capability of Obstacle Detection
The system can be used in 2 specific situations: in walking mode the system is detecting hazards when the user is walking, while in exploration mode the user is stationary and sweeps slowly to scan the scene. The Table 3 gives a summary of the obstacle detection capabilities in those situations or modes.
Figure 1 illustrates a 2D view of the two dead zones, and Table 4 shows examples about the maximum length of the two dead zones when with different users’ tall and different height of the mounted camera.
2. Gap analysis to exploitation
At the end of the project, the involved SME’s have reviewed the efforts required to make a product out of the RANGE-IT prototype. This review is based upon the final D1.2 which contains of features and issues of the prototype. This list contains new requirements that where not foreseen, bugs that where identified too late to be solved within the project, or requirements left out of scope deliberately.
All SME’s agree that there are still open limitations and remaining research to be resolved before product launch. Most of these issues became apparent at the user tests and findings of the project partners in the final stages of the project. The updated expectations of the SME’s and corresponding gap analysis relates mostly to the user experience of the system that requires the reduction of information from the visual input towards the tactile and audio feedback. A follow-up project with guidance from a psychology expert is hence desirable to understand better the amount of information that can be handled by the users of a potential product.
The table 5 sums up the open issues of development aspects and the scientific ICT and psychology research towards a product launch.
3. Forground resulting from the project
As by default in FP7 projects and in accordance to definitions and guidelines provided in the "Guide to Intellectual Property Rules for FP7 projects" published in CORDIS (http://ec.europa.eu/research/participants/data/ref/fp7/89593/ipr_en.pdf) the consortium handles IPR according to the following principles:
1. Foreground IP ownership lies with the SME participants, RTD participants are remunerated for their efforts.
2. Foreground IP ownership lies with the SME responsible for the generation of such IP, be it through own effort or subcontracting to an RTD participant.
3. Knowledge generated by joint work of several SME partners where respective shares of the work cannot be ascertained, shall be jointly owned. The SME parties concerned shall agree to the allocation and terms of exercising ownership (otherwise a default regime for joint ownership is foreseen).
4. All access rights to background and foreground IP for project execution and use purposes within the project shall be royalty-free to and by all partners. In particular, during the exploitation phase the SMEs will not pay any royalties for access to the background of the RTD performers (if necessary). Ownership of background IP is never affected.
5. RTD participants will not have any ownership or other user rights on foreground IP, but may request access rights for internal research purposes. The SMEs may grant such access rights provided that the granting is consistent with their commercial interests. Such access rights shall be granted to the RTD performers on fair and reasonable conditions to be agreed on by the SMEs on a case by-case basis.
6. Side-ground knowledge (acquired in parallel to the contract) is to be negotiated between partners on a case-by-case if access for the project is needed.
Furthermore, the standard rules in the EC contract have been implemented without restriction. The specific RANGE-IT CA, has clear some complementary points in order to avoid any misunderstandings and will identify all background knowledge needed at the start of the project.
Patenting efforts on foreground during the project can be summarized as follows:
• SKS is in the process of filing a patent on the depth & point cloud filters (as described in D2.2). The garment and the remote button system have been placed in i-depot, in order to strengthen the intellectual property position in potential future patent matters.
• Elitac has filed a patent during the project on their manufacturing method for garments. This method is not a result of efforts in the project, but does protect the IP of all garment designs produced by Elitac.
• D&L has not made a patent nor product in the direction of the work in RANGE-IT, but has gained expertise on visual and tactile capabilities.
The Table 6 describes the exploitable foreground (for the SME), and Table 7 links this to the workpackages
The SMEs decided it is not in their strategic interest to apply for a joint patent on the RANGE-IT system. Their commercial interests are covered in their respective IP strategies for the subsystems. Any development contributions to the matters listed in the GAP analysis would be welcomed by the SMEs, and do not need to be proprietary. Therefore, it was decided that the RTD’s can use their respective work and the general foreground as a basis for future research, without conflict with the commercial interests of the SMEs.
3. IP
3.1 Patents outside the consortium
A thorough patent search has been carried out at the initiation of the concept. Several patent applications have been found related to navigation using tactile feedback, either targeting blind people or not specifying a target group. However, most of these patents have either expired or have never been issued:
• EP1035463 - Device for indicating relative location to a user;
• EP0774245 - Orientation aid for the visually impaired;
• US 2008/120029 - Wearable tactile navigation system;
• PCT/US97/14525 - An Electronic auto-routing navigation system for visually impaired persons.
Only one patent related to RANGE-IT is in effect in several countries in the EU: EP1220179 - Navigation system. However, unlike RANGE-IT, this patent does not use any knowledge about the environment, but merely uses the user’s location, and possibly his orientation and destination. Additionally, no auditory feedback is provided either.
In the USA, two more patents related to his project are currently in effect:
• US7788032 - Targeting Location Through Haptic Feedback Signals;
• US6424333 - Tactile feedback man-machine interface device.
The first of these patents, with patent number US7788032, does not use any environmental knowledge, similar to the patent described in the previous paragraph.
Although the second patent, US6424333, does describe a general tactile feedback mechanism that could be similar to the tactile feedback used in the RANGE-IT system, this patent only considers its use in a simulated environment.
Finally, several patents on navigation using beacons exist, such as US6320496 - Systems and methods providing tactile guidance using sensory supplementation. However, since RANGE-IT does not make use of beacons, these patents are not applicable to the project.
However patents related to blind navigation exist, an application like RANGE-IT has not been patented. The concepts of the system’s parts have been checked also. Therefore, at the start of the RANGE-IT project, patent searches have proven that no intellectual property rights will be infringed. No additional patent scan was performed during the project on the RANGE-IT system aspects because of the identified gap towards product launch, as discussed in section 2.
During the project the partners continued to verify if no IP related to RANGE-IT has been put on the market. To our best knowledge, all related products and research outside the consortium is described in section B2 of the PUDF (deliverable 6.2) . The market search has proven that no commercial product of the characteristics of RANGE-IT exists yet.
3.2 Foreground resulting from the project
As by default in FP7 projects and in accordance to definitions and guidelines provided in the "Guide to Intellectual Property Rules for FP7 projects" published in CORDIS (http://ec.europa.eu/research/participants/data/ref/fp7/89593/ipr_en.pdf) the consortium handles IPR according to the following principles:
1. Foreground IP ownership lies with the SME participants, RTD participants are remunerated for their efforts.
2. Foreground IP ownership lies with the SME responsible for the generation of such IP, be it through own effort or subcontracting to an RTD participant.
3. Knowledge generated by joint work of several SME partners where respective shares of the work cannot be ascertained, shall be jointly owned. The SME parties concerned shall agree to the allocation and terms of exercising ownership (otherwise a default regime for joint ownership is foreseen).
4. All access rights to background and foreground IP for project execution and use purposes within the project shall be royalty-free to and by all partners. In particular, during the exploitation phase the SMEs will not pay any royalties for access to the background of the RTD performers (if necessary). Ownership of background IP is never affected.
5. RTD participants will not have any ownership or other user rights on foreground IP, but may request access rights for internal research purposes. The SMEs may grant such access rights provided that the granting is consistent with their commercial interests. Such access rights shall be granted to the RTD performers on fair and reasonable conditions to be agreed on by the SMEs on a case by-case basis.
6. Side-ground knowledge (acquired in parallel to the contract) is to be negotiated between partners on a case-by-case if access for the project is needed.
Furthermore, the standard rules in the EC contract have been implemented without restriction. The specific RANGE-IT CA, has clear some complementary points in order to avoid any misunderstandings and identified all background knowledge needed at the start of the project.
Patenting efforts on foreground during the project can be summarized as follows:
• SKS is in the process of filing a patent on the depth & point cloud filters (as described in D2.2 of the PUDF). The garment and the remote button system have been placed in i-depot, in order to strengthen the intellectual property position in potential future patent matters.
• Elitac has filed a patent during the project on their manufacturing method for garments. This method is not a result of efforts in the project, but does protect the IP of all garment designs produced by Elitac.
• D&L has not made a patent nor product in the direction of the work in RANGE-IT, but has gained expertise on visual and tactile capabilities.
• The SMEs decided it is not in their strategic interest to apply for a joint patent on the RANGE-IT system. Their commercial interests are covered in their respective IP strategies for the subsystems. Any development contributions to the matters listed in the GAP analysis would be welcomed by the SMEs, and do not need to be proprietary. Therefore, it was decided that the RTD’s can use their respective work and the general foreground as a basis for future research, without conflict with the commercial interests of the SMEs.
Potential Impact:
Impact on blind users
From a humanitarian and economical point of view, it makes sense to equip blind persons with a technical device once rather than paying for assistance carried out by individuals on a regular basis. There are no complete alternatives to the RANGE-IT system, although there is some overlap with the use of a guide dog or a human assistant. From a financial perspective, the RANGE-IT system offers great value for money to the customer. At an intended sales price of around 1400 euro (excluding a smartphone) the investment is significantly lower than that of a guide dog (approx. 30.000 euro), and virtually no operational costs are incurred. Obviously, the costs of a human assistant are even higher.
Range-It is a system that enables blind persons to explore their environment beyond the range of the cane. Such products do not exist. Range-It does not substitute vision but offers tools that enable blind people to get specific information about their environment enabling them to switch between the different modes and filters in order to get the specific information needed in the current context. Wherever independent orientation and mobility for blind people is needed the product can be used. This includes private and vocational use.
Impact on SMEs
For an SME, a project like RANGE-IT opens up the possibility for the development of assistive technology for a relatively small market (about 2 % of the population are visually impaired or blind). Enabling blind users to explore their environment and to find their way without assistance by sighted persons or by less assistance creates an economic value, as services for exploration and guidance are being carried out by technical systems rather than by individuals.
The creation of a new device for independent exploration of an indoor environment opens up a new market for the SMEs. It also improves their competitiveness, as the device can be used in conjunction with other technology opening up synergy effects.
Dissemination
The dissemination strategy comprises the way the consortium communicates with the public, and especially the blind community, regarding the project itself, its activities and their outcome. The dissemination is intended to reach the principal worldwide stakeholders in the health device builder ecosystem.
From the beginning of the project, dissemination was organized towards the end-users community, namely the health sector (especially the blind users communities), members of organizations (executives, managers and other relevant personnel in organizations of all kinds – private and public), policy makers and other influential persons or organization. The promotion was carried out by presentations of the RANGE-IT public results in events, conferences, workshops and publications in scientific and professional journals. Meanwhile, a press release (D6.1) and a public website of the project (http://range-it.eu/ hosted at SKS, maintained by UGent) , as well as a project video (D6.3) are available.
The attached list with dissemination activities mentions dissemination activities by
- the dissemination date (when it was/will be disseminated),
- the type of the activity (how it was/will be disseminated),
- the place of the activity (where it was/will be disseminated),
- the target audience (to who it was/will be disseminated),
- the lead partner and contact (who was/will be disseminating),
- the title (descriptive title, as was/will be disseminated)
These dissemination activities have been, whenever possible, archived on the public website.
Since the nature of the ‘Research for the benefit of SME’ project is not to practice fundamental research, but is aiming at higher TRL levels, there is no specific focus on publications, although the partners had the possibility to do so (as mentioned in the project’s Consortium Agreement). Note that especially TUDresden was very reluctant to reveal the details of the system as we were instructed to do so during the project, for example the exploration mode nowhere described in the publications.
Furthermore, as part of the good practices in the management of the Range-it project, we ‘d like to stress that for each dissemination activity (and each deliverable) the impacted SME partners have been informed and have reviewed the RTD contributions.
In particular, besides the activities mentioned in the dissemination document, an outreach to the blind community has been initiated by the partners (especially by the RTD) to organize user tests (as part of WP1, WP4 and WP5). For a description of these specific activities and their results, we refer to the submitted project confidential deliverables of these WP.
The following meetings took place with members of the blind community:
TUD had its main activities related to WP4 and WP5:
• Demo to blind users at SightCity 2015, with D&L,
• Make a tutorials at ICC workshop 2015 by Prof. Weber,
• Evaluation with blind users in Zeist
TNO had its contribution in the framework of WP5:
Evaluation with blind users at Zeist
• August 2015: Meeting with Marten van Doorn (Bartimeus), introduction of project
• 24 september 2015: Meeting with Marten van Doorn (Bartimeus) and Eric Velleman (Stichting Accessibility), introduction and discussion of project.
UGent performed the following actions in Belgium as part of their work in WP1:
• 12/11/2013: Meeting Mia Engels from Transkript to recruit participants
• 13/11/2013: Meeting Marie-Jeanne Nachtergaele, Brailleliga to recruit participants
• 22/11/2013: Meeting Brailleliga Kortrijk to introduce project
• 29/01/2014: Meeting Johan de Mol (Department of Telecommunications andinformation processing), Gerry Guldentops to introduce project
• 06/02/2014: Presentation Project Braileliga Ghent and interviews
• 14/22/2014: Discussion and presentation at Revalidation centre De Markgrave in Antwerp
• 02/02/2015: Discussion at Spermalie in Brugge
List of Websites:
http://range-it.eu/
For the Millions of people who are blind or visually impaired, moving around in dynamic indoor environments can be very challenging. Their perception of the world usually ends as far as their white cane reach.
Multiple Electronic Travel Aids (ETA) are available as assistive technologies, which help to avoid obstacles and hazards. Many of these early ETAs use ultrasonic sensors or digital cameras to detect obstacles.
Most of those existing prototypes and commercial products only present whether there are obstacles on their way, however, they fail to provide a comprehensive situation awareness function which let blind and visually impaired people better understand their surrounding environment.
The 3 most important problems the current version of the Range-IT system aims to solve are:
1. Spatial awareness of visually impaired people ends at the end of the white cane
2. Current products do not provide a comprehensive situational awareness to understand their surrounding environment.
3. Audio based systems interfere with the main information channel available to the visually impaired.
The Range-IT system allows blind and visually impaired users to understand the surrounding indoor environment. A 3D time-of-flight camera scans the environment for stairs, obstacles (hanging, standing or moving). The camera is able to detect a range up to 5m which significantly enlarges the spatial awareness of the visually impaired in comparison to a white cane.
The Range-IT system can;
- Detect the existence of obstacles ahead of the user beyond the reach of the cane up to 5 meters when walking
- Detect the existence of obstacles in a virtual corridor left and right.
- Detect hanging obstacles that can cause collisions.
- Support users in finding stairs that lead upwards
- Support users in finding open doors
- Support users in getting a basic picture of the structure of a room
- Support users in getting an idea about the distance to a wall and its orientation.
The user is informed about the different obstacles by means of tactile (vibration) patterns and by interactive spatialized sounds. A big advantage of tactile representation is that the user can hear environmental sounds and obtain information about obstacles simultaneously. Relevant information can be given via small vibrations motors on the belt, without overloading the user. Through the spatialized signification sounds, blind and visually impaired users are able to quickly acquire direction and distance information surrounding obstacles.
Project Context and Objectives:
The RANGE-IT system is a stand-alone wearable prototype for the blind and visually impaired, which detects and presents close range standing, and hanging objects to the user in indoor environments. Furthermore, specific reference points (e.g. stairs, walls and open doors) can be identified. Information is presented in tactile and/or audio modality based on the context.
The market for Assistive Technology (AT) addresses a growing number of European citizens, as demographics change and more and more people grow older. Economics in this domain consists of a mixture of private and public sources and is bound to specific activities including education, work and daily living. For example blind or visually impaired people are often supported by public funding regarding the purchase of technological assistance systems, which shows the public concern to support activities necessary for an independent life, such as reading printed documents through scanners and speech synthesizers. The World Health Organisation states that one of 175 people is totally blind (i.e. visually impaired people are not included). Although there is some assistive technology available for blind people at the work place and at home, this doesn’t include assistance in mobility in other places. The majority of blind people are only mobile if they have help of others (in person or by mobile phone), or use solely known pace ways. Besides the cane with a limited range and an expensive guide dog, there are very few aids that support individual mobility of the blind.
A product aiming at mobility addresses not only people at the workplace but may reach all blind people, if they can afford it or see its cost justified. If compared to the functionality of a colour checker (a device naming colours) the price will be slightly higher but many more benefits will be gained. Moreover, as a ramp technology, it is expected also people with low vision want to feel safer in unknown environments through the RANGE-IT system.
The prototype of the RANGE-IT system is developed using iterative and parallel research determining the multiplicity of user requirements in different contexts and matching it with device and user capabilities. With regard to user capabilities, The RANGE-IT system will for example add only little additional effort to the already high demand on the blind person to observe the environment through hearing sound scenes and touching the ground by feet.
The white cane can detect the ground surface including changes in the ground texture and it emits sound allowing the walker to monitor changes in the environment such as bushes, walls and openings. It cannot detect obstacles above the waist and nothing beyond the tip of the cane. The reachable space (range) of an arm and cane by turning around is about 3.14 m2 (1 m x 1 m x 3.14). RANGE-IT extends this range to 5m, . In the large detectable area, multiple obstacles can be captured and located simultaneously. Thereby, it is possible to acquire the overview layout of surrounded obstacles.
Compared to the conventional strategy of a blind person to "scan" the inside of a huge shopping mall by walking around and performing trial and error movements, the number of necessary trials can be reduced drastically by extending the range. Furthermore, during mobility training, blind people learn how to move along house walls, fences, pedestrian walks, etc. These structures create division lines. Unlike sighted people who perceive division lines visually while moving along them, blind people re-create them from auditory and tactile perception along their own movement. RANGE-IT represents those structures in a virtual and accessible way without the need for the blind person to move all the way along them.
In sum, the objectives of the RANGE-IT system are:
• Extension of range of perception. RANGE-IT has a range of up to 5 meters, which will be a world of difference to the user.
• Empowerment. RANGE-IT will empower visually impaired people to act more independently in society.
• Low cost. The RANGE-IT system will be low cost given the high accuracy. There is no comparable aid on the market.
• Independence. Visually impaired people can act more spontaneously and independently. For example the following activities can be performed more independently: general mobility; visiting authorities; shopping; using public transport.
• Assistance. Less (human) assistance is needed.
• Participation. Mobility is connected to taking part in social life. Increasing mobility thus supports social participation.
Project Results:
1. Specification of the Range-IT system
For the design steps of the system we refer to the confidential deliverables in WP1, and for the detailed results to the confidential deliverables in WP2 to WP5. An overview of the system and of the capabilities of the obstacle detection is given here.
1.1 Overview of the Range-IT System
1. Main Hardware Components:
a. A Softkinetic infrared time-of-flight 3D camera
i. Size (W x H x D): 105 x 30 x 23 mm
ii. Power: < 2.5 w
iii. Illumination type: Diffused laser, For indoor use
iv. Range: 0.15 – 7 m (short range: 0.15 – 1.0m)
v. Resolution: 320 x 240
vi. Frame Rate: 25 – 60 fps
vii. Field of view (H x V x D): 74° x 58° x 87°
b. An ARM-based ODROID-U3 embedded board
i. 1.7GHz Quad-Core CPU
ii. 2GByte RAM
c. An Elitac tactile belt with multiple vibratos
d. A bone-conduction earphone
e. A wired controller with 3 keys
f. A common 5V Li-ion battery with USB output
g. A specific designed harness
2. Power consumption:
a. 5V input
b. Max. 2.9 A
c. System runtime for 6 hours with a full 20Ah 5V Li-ion battery
3. Weight: 1.5kg (including the backpack)
1.2. Capability of Obstacle Detection
The system can be used in 2 specific situations: in walking mode the system is detecting hazards when the user is walking, while in exploration mode the user is stationary and sweeps slowly to scan the scene. The Table 3 gives a summary of the obstacle detection capabilities in those situations or modes.
Figure 1 illustrates a 2D view of the two dead zones, and Table 4 shows examples about the maximum length of the two dead zones when with different users’ tall and different height of the mounted camera.
2. Gap analysis to exploitation
At the end of the project, the involved SME’s have reviewed the efforts required to make a product out of the RANGE-IT prototype. This review is based upon the final D1.2 which contains of features and issues of the prototype. This list contains new requirements that where not foreseen, bugs that where identified too late to be solved within the project, or requirements left out of scope deliberately.
All SME’s agree that there are still open limitations and remaining research to be resolved before product launch. Most of these issues became apparent at the user tests and findings of the project partners in the final stages of the project. The updated expectations of the SME’s and corresponding gap analysis relates mostly to the user experience of the system that requires the reduction of information from the visual input towards the tactile and audio feedback. A follow-up project with guidance from a psychology expert is hence desirable to understand better the amount of information that can be handled by the users of a potential product.
The table 5 sums up the open issues of development aspects and the scientific ICT and psychology research towards a product launch.
3. Forground resulting from the project
As by default in FP7 projects and in accordance to definitions and guidelines provided in the "Guide to Intellectual Property Rules for FP7 projects" published in CORDIS (http://ec.europa.eu/research/participants/data/ref/fp7/89593/ipr_en.pdf) the consortium handles IPR according to the following principles:
1. Foreground IP ownership lies with the SME participants, RTD participants are remunerated for their efforts.
2. Foreground IP ownership lies with the SME responsible for the generation of such IP, be it through own effort or subcontracting to an RTD participant.
3. Knowledge generated by joint work of several SME partners where respective shares of the work cannot be ascertained, shall be jointly owned. The SME parties concerned shall agree to the allocation and terms of exercising ownership (otherwise a default regime for joint ownership is foreseen).
4. All access rights to background and foreground IP for project execution and use purposes within the project shall be royalty-free to and by all partners. In particular, during the exploitation phase the SMEs will not pay any royalties for access to the background of the RTD performers (if necessary). Ownership of background IP is never affected.
5. RTD participants will not have any ownership or other user rights on foreground IP, but may request access rights for internal research purposes. The SMEs may grant such access rights provided that the granting is consistent with their commercial interests. Such access rights shall be granted to the RTD performers on fair and reasonable conditions to be agreed on by the SMEs on a case by-case basis.
6. Side-ground knowledge (acquired in parallel to the contract) is to be negotiated between partners on a case-by-case if access for the project is needed.
Furthermore, the standard rules in the EC contract have been implemented without restriction. The specific RANGE-IT CA, has clear some complementary points in order to avoid any misunderstandings and will identify all background knowledge needed at the start of the project.
Patenting efforts on foreground during the project can be summarized as follows:
• SKS is in the process of filing a patent on the depth & point cloud filters (as described in D2.2). The garment and the remote button system have been placed in i-depot, in order to strengthen the intellectual property position in potential future patent matters.
• Elitac has filed a patent during the project on their manufacturing method for garments. This method is not a result of efforts in the project, but does protect the IP of all garment designs produced by Elitac.
• D&L has not made a patent nor product in the direction of the work in RANGE-IT, but has gained expertise on visual and tactile capabilities.
The Table 6 describes the exploitable foreground (for the SME), and Table 7 links this to the workpackages
The SMEs decided it is not in their strategic interest to apply for a joint patent on the RANGE-IT system. Their commercial interests are covered in their respective IP strategies for the subsystems. Any development contributions to the matters listed in the GAP analysis would be welcomed by the SMEs, and do not need to be proprietary. Therefore, it was decided that the RTD’s can use their respective work and the general foreground as a basis for future research, without conflict with the commercial interests of the SMEs.
3. IP
3.1 Patents outside the consortium
A thorough patent search has been carried out at the initiation of the concept. Several patent applications have been found related to navigation using tactile feedback, either targeting blind people or not specifying a target group. However, most of these patents have either expired or have never been issued:
• EP1035463 - Device for indicating relative location to a user;
• EP0774245 - Orientation aid for the visually impaired;
• US 2008/120029 - Wearable tactile navigation system;
• PCT/US97/14525 - An Electronic auto-routing navigation system for visually impaired persons.
Only one patent related to RANGE-IT is in effect in several countries in the EU: EP1220179 - Navigation system. However, unlike RANGE-IT, this patent does not use any knowledge about the environment, but merely uses the user’s location, and possibly his orientation and destination. Additionally, no auditory feedback is provided either.
In the USA, two more patents related to his project are currently in effect:
• US7788032 - Targeting Location Through Haptic Feedback Signals;
• US6424333 - Tactile feedback man-machine interface device.
The first of these patents, with patent number US7788032, does not use any environmental knowledge, similar to the patent described in the previous paragraph.
Although the second patent, US6424333, does describe a general tactile feedback mechanism that could be similar to the tactile feedback used in the RANGE-IT system, this patent only considers its use in a simulated environment.
Finally, several patents on navigation using beacons exist, such as US6320496 - Systems and methods providing tactile guidance using sensory supplementation. However, since RANGE-IT does not make use of beacons, these patents are not applicable to the project.
However patents related to blind navigation exist, an application like RANGE-IT has not been patented. The concepts of the system’s parts have been checked also. Therefore, at the start of the RANGE-IT project, patent searches have proven that no intellectual property rights will be infringed. No additional patent scan was performed during the project on the RANGE-IT system aspects because of the identified gap towards product launch, as discussed in section 2.
During the project the partners continued to verify if no IP related to RANGE-IT has been put on the market. To our best knowledge, all related products and research outside the consortium is described in section B2 of the PUDF (deliverable 6.2) . The market search has proven that no commercial product of the characteristics of RANGE-IT exists yet.
3.2 Foreground resulting from the project
As by default in FP7 projects and in accordance to definitions and guidelines provided in the "Guide to Intellectual Property Rules for FP7 projects" published in CORDIS (http://ec.europa.eu/research/participants/data/ref/fp7/89593/ipr_en.pdf) the consortium handles IPR according to the following principles:
1. Foreground IP ownership lies with the SME participants, RTD participants are remunerated for their efforts.
2. Foreground IP ownership lies with the SME responsible for the generation of such IP, be it through own effort or subcontracting to an RTD participant.
3. Knowledge generated by joint work of several SME partners where respective shares of the work cannot be ascertained, shall be jointly owned. The SME parties concerned shall agree to the allocation and terms of exercising ownership (otherwise a default regime for joint ownership is foreseen).
4. All access rights to background and foreground IP for project execution and use purposes within the project shall be royalty-free to and by all partners. In particular, during the exploitation phase the SMEs will not pay any royalties for access to the background of the RTD performers (if necessary). Ownership of background IP is never affected.
5. RTD participants will not have any ownership or other user rights on foreground IP, but may request access rights for internal research purposes. The SMEs may grant such access rights provided that the granting is consistent with their commercial interests. Such access rights shall be granted to the RTD performers on fair and reasonable conditions to be agreed on by the SMEs on a case by-case basis.
6. Side-ground knowledge (acquired in parallel to the contract) is to be negotiated between partners on a case-by-case if access for the project is needed.
Furthermore, the standard rules in the EC contract have been implemented without restriction. The specific RANGE-IT CA, has clear some complementary points in order to avoid any misunderstandings and identified all background knowledge needed at the start of the project.
Patenting efforts on foreground during the project can be summarized as follows:
• SKS is in the process of filing a patent on the depth & point cloud filters (as described in D2.2 of the PUDF). The garment and the remote button system have been placed in i-depot, in order to strengthen the intellectual property position in potential future patent matters.
• Elitac has filed a patent during the project on their manufacturing method for garments. This method is not a result of efforts in the project, but does protect the IP of all garment designs produced by Elitac.
• D&L has not made a patent nor product in the direction of the work in RANGE-IT, but has gained expertise on visual and tactile capabilities.
• The SMEs decided it is not in their strategic interest to apply for a joint patent on the RANGE-IT system. Their commercial interests are covered in their respective IP strategies for the subsystems. Any development contributions to the matters listed in the GAP analysis would be welcomed by the SMEs, and do not need to be proprietary. Therefore, it was decided that the RTD’s can use their respective work and the general foreground as a basis for future research, without conflict with the commercial interests of the SMEs.
Potential Impact:
Impact on blind users
From a humanitarian and economical point of view, it makes sense to equip blind persons with a technical device once rather than paying for assistance carried out by individuals on a regular basis. There are no complete alternatives to the RANGE-IT system, although there is some overlap with the use of a guide dog or a human assistant. From a financial perspective, the RANGE-IT system offers great value for money to the customer. At an intended sales price of around 1400 euro (excluding a smartphone) the investment is significantly lower than that of a guide dog (approx. 30.000 euro), and virtually no operational costs are incurred. Obviously, the costs of a human assistant are even higher.
Range-It is a system that enables blind persons to explore their environment beyond the range of the cane. Such products do not exist. Range-It does not substitute vision but offers tools that enable blind people to get specific information about their environment enabling them to switch between the different modes and filters in order to get the specific information needed in the current context. Wherever independent orientation and mobility for blind people is needed the product can be used. This includes private and vocational use.
Impact on SMEs
For an SME, a project like RANGE-IT opens up the possibility for the development of assistive technology for a relatively small market (about 2 % of the population are visually impaired or blind). Enabling blind users to explore their environment and to find their way without assistance by sighted persons or by less assistance creates an economic value, as services for exploration and guidance are being carried out by technical systems rather than by individuals.
The creation of a new device for independent exploration of an indoor environment opens up a new market for the SMEs. It also improves their competitiveness, as the device can be used in conjunction with other technology opening up synergy effects.
Dissemination
The dissemination strategy comprises the way the consortium communicates with the public, and especially the blind community, regarding the project itself, its activities and their outcome. The dissemination is intended to reach the principal worldwide stakeholders in the health device builder ecosystem.
From the beginning of the project, dissemination was organized towards the end-users community, namely the health sector (especially the blind users communities), members of organizations (executives, managers and other relevant personnel in organizations of all kinds – private and public), policy makers and other influential persons or organization. The promotion was carried out by presentations of the RANGE-IT public results in events, conferences, workshops and publications in scientific and professional journals. Meanwhile, a press release (D6.1) and a public website of the project (http://range-it.eu/ hosted at SKS, maintained by UGent) , as well as a project video (D6.3) are available.
The attached list with dissemination activities mentions dissemination activities by
- the dissemination date (when it was/will be disseminated),
- the type of the activity (how it was/will be disseminated),
- the place of the activity (where it was/will be disseminated),
- the target audience (to who it was/will be disseminated),
- the lead partner and contact (who was/will be disseminating),
- the title (descriptive title, as was/will be disseminated)
These dissemination activities have been, whenever possible, archived on the public website.
Since the nature of the ‘Research for the benefit of SME’ project is not to practice fundamental research, but is aiming at higher TRL levels, there is no specific focus on publications, although the partners had the possibility to do so (as mentioned in the project’s Consortium Agreement). Note that especially TUDresden was very reluctant to reveal the details of the system as we were instructed to do so during the project, for example the exploration mode nowhere described in the publications.
Furthermore, as part of the good practices in the management of the Range-it project, we ‘d like to stress that for each dissemination activity (and each deliverable) the impacted SME partners have been informed and have reviewed the RTD contributions.
In particular, besides the activities mentioned in the dissemination document, an outreach to the blind community has been initiated by the partners (especially by the RTD) to organize user tests (as part of WP1, WP4 and WP5). For a description of these specific activities and their results, we refer to the submitted project confidential deliverables of these WP.
The following meetings took place with members of the blind community:
TUD had its main activities related to WP4 and WP5:
• Demo to blind users at SightCity 2015, with D&L,
• Make a tutorials at ICC workshop 2015 by Prof. Weber,
• Evaluation with blind users in Zeist
TNO had its contribution in the framework of WP5:
Evaluation with blind users at Zeist
• August 2015: Meeting with Marten van Doorn (Bartimeus), introduction of project
• 24 september 2015: Meeting with Marten van Doorn (Bartimeus) and Eric Velleman (Stichting Accessibility), introduction and discussion of project.
UGent performed the following actions in Belgium as part of their work in WP1:
• 12/11/2013: Meeting Mia Engels from Transkript to recruit participants
• 13/11/2013: Meeting Marie-Jeanne Nachtergaele, Brailleliga to recruit participants
• 22/11/2013: Meeting Brailleliga Kortrijk to introduce project
• 29/01/2014: Meeting Johan de Mol (Department of Telecommunications andinformation processing), Gerry Guldentops to introduce project
• 06/02/2014: Presentation Project Braileliga Ghent and interviews
• 14/22/2014: Discussion and presentation at Revalidation centre De Markgrave in Antwerp
• 02/02/2015: Discussion at Spermalie in Brugge
List of Websites:
http://range-it.eu/