CORDIS

EU research results

CORDIS

Spatial-temporal information processing for collision detection in dynamic environments

Fact Sheet

Reporting

Results

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

United Kingdom

Objective

In the real world, collision happens at every second - often results in serious accidents and fatalities. For example, there are more than 3560 people died from vehicle collision per day worldwide. On the other sector, autonomous unmanned aerial vehicles (UAVs) have demonstrated great potential in serving human society such as delivering goods to households and precision farming, but are restricted due to lacking of collision detection capability. The current approaches for collision detection such as radar, laser based Ladar and GPS are far from acceptable in terms of reliability, energy consumption and size. A new type of low cost, low energy consumption and miniaturized collision detection sensors are badly needed to not only save millions of people’s lives but also make autonomous UAVs and robots safe to serve human society. STEP2DYNA consortium proposes an innovative bio-inspired solution for collision detection in dynamic environments. It takes the advantages of low cost spatial-temporal and parallel computing capacity of visual neural systems and realized it in chip specifically for collision detection in dynamic environments.

Realizing visual neural systems in chips demands multidisciplinary expertise in biological system modelling, computer vision, chip design and robotics. This breadth of expertise is not readily possessed within one institution. Secondly, the market potential of the collision detection system could not be well exploited, unless by a dedicated partner from industry. Therefore, this consortium is designed to bring neurobiologists, neural system modelers, chip designers, robotics researchers and engineers from Europe and East of Asia together and complement each others’ research strengths via staff secondments, jointly organised workshops and conferences. Through this project, the partners will build up strong expertise in this exciting multidisciplinary area and the European SME will position well as a market leader in collision detection.

Field of Science

/social sciences/economics and business/business and management/commerce

/natural sciences/computer and information sciences/artificial intelligence/computer vision

/social sciences/psychology/cognitive psychology

/natural sciences/computer and information sciences/data science/data processing

/engineering and technology/electrical engineering, electronic engineering, information engineering/electronic engineering/robotics

Programme(s)

H2020-EU.1.3.3. - Stimulating innovation by means of cross-fertilisation of knowledge

Topic(s)

MSCA-RISE-2015 - Marie Skłodowska-Curie Research and Innovation Staff Exchange (RISE)

Call for proposal

H2020-MSCA-RISE-2015

See other projects for this call

Funding Scheme

MSCA-RISE - Marie Skłodowska-Curie Research and Innovation Staff Exchange (RISE)

Coordinator

UNIVERSITY OF LINCOLN

Address

Brayford Pool
Ln6 7ts Lincoln

United Kingdom

Activity type

Higher or Secondary Education Establishments

EU Contribution

€ 517 500

Website

Contact the organisation

Participants (3)

Sort alphabetically

Sort by EU Contribution

Expand all

UNIVERSITAET HAMBURG

Germany

EU Contribution

€ 121 500

UNIVERSITY OF NEWCASTLE UPON TYNE

United Kingdom

EU Contribution

€ 243 000

AGILE ROBOTS AG

Germany

EU Contribution

€ 126 000

Partners (7)

Sort alphabetically

Expand all

TSINGHUA UNIVERSITY

China

NATIONAL UNIVERSITY CORPORATION KYUSHU UNIVERSITY

Japan

XI'AN JIAOTONG UNIVERSITY

China

Huazhong University of Science and Technology

China

UNIVERSIDAD DE BUENOS AIRES

Argentina

Guangzhou University

China

UNIVERSITI PUTRA MALAYSIA

Malaysia

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

United Kingdom

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

United Kingdom

Periodic Reporting for period 1 - STEP2DYNA (Spatial-temporal information processing for collision detection in dynamic environments)

Reporting period: 2016-07-01 to 2018-06-30

Summary of the context and overall objectives of the project

In the real world, collisions happens at every second - often resulting in serious accidents and fatalities. For example, more than 3560 people die from vehicle collision per day worldwide. On the other sector, autonomous unmanned aerial vehicles (UAVs) have demonstrated great potential in serving human society such as delivering goods to households and precision farming, but are restricted due to lacking of collision detection capability. The current approaches for collision detection such as radar, laser based Ladar and GPS are far from acceptable in terms of reliability, energy consumption and size. A new type of low cost, low energy consumption and miniaturized collision detection sensors are urgently needed to not only save millions of people’s lives, but also make autonomous UAVs and robots safe to serve human society.

The STEP2DYNA consortium proposes an innovative bio-inspired solution for collision detection in dynamic environments. It takes the advantages of low cost spatial-temporal and parallel computing capacity of visual neural systems and realizes it in chips specifically for collision detection in dynamic
environments. Realizing visual neural systems in chips demands multidisciplinary expertise in biological system modelling, computer vision, chip design and robotics. This breadth of expertise is not readily possessed within one institution. Secondly, the market potential of the collision detection system could not be well exploited, unless by a dedicated partner from industry.

Therefore, this consortium is designed to bring neurobiologists, neural system modellers, chip designers, robotics researchers and engineers from Europe and the East of Asia together and complement each other’s research strengths via staff secondments, and jointly organised workshops and conferences. Through this project, the partners will build strong expertise in this exciting multidisciplinary area and the European SME will position well as a market leader in collision detection sensors.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The first work package of the project focusses on modelling, testing and comparing visual neural systems. Researchers have pioneered a novel collision selective visual neural network inspired by a specific group of lobula giant movement detectors (LGMD) neurons on the juvenile locust.

In terms of model comparison and selection, researchers have studied suitable models and proposed a bio-plausible model, the angular velocity detecting model, for estimating the image motion velocity using the latest neural circuit’s discoveries of the Drosophila visual systems.

The second work package focuses on multiple neural systems, which are needed to enhance the reliability and robustness of the system. Researchers have proposed a directional selective based neural network for small target detection in a cluttered background. The proposed neural network not only can detect small target motion but it can also specify the direction of small target motion in cluttered backgrounds. Extensive experiments showed that the proposed neural network can reliably detect small targets against cluttered backgrounds.

Work package 3 relates to the identification and selection of the best model for chip realisation. This involves identifying and comparing specialised neural models for collision detection, and investigating design factors such as feasibility, suitability and production costs. The identified neural systems will be tested under more realistic electronic noise after the main chip structure has been identified, and finally the VLSI (very-large-scale integration) chip will be further designed and tested.

Researchers have identified the Lobula Giant Movement Detector (LGMD1) as the most reliable model, and have implemented it on to a UAV demonstrating the model’s preliminary ability for collision avoidance.

Progress has been made with the chip design and this is a key piece of work. The work has included measurement of the JESD204B chip and design of the RILCM chips for the injection mechanism analysis. The JESD204B chip design is aimed to widen the lane number of the data communication for the VLSI chip.

The final technical work package looks at robots being equipped with visual sensors. The preliminary visual neural system will be selected and implemented on a robotic platform, with a motor control system for collision detection and avoidance. This will then be implemented onto a UAV platform.

Researchers have implemented the lobula giant movement detector LGMD1 on to a UAV demonstrating the preliminary ability for collision avoidance. This has involved building a multi sensor quadcopter platform, selecting the hardware and software design, designing software on the computer, carrying out flight experiments and writing a paper for the application of the LGMD on the quadcopter. The quadcopter is a multi-sensor platform which can provide posture, height, direction, and position information and provide a command interface for the LGMD vision-based collision detector.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The modelling work is ongoing. Vision is a necessary cue for human beings to detect collision, but the principle between vision and collision detection is still unknown. The basic component of vision is gaze, which can be calculated by eye gaze tracking. However, current 3D gaze tracking in the real environment is limited by the accuracy. Researchers aim to develop a high accurate eye tracker to estimate human gaze by employing more accurate projective models.

Further modelling work will be to research a collision detecting model and integrate it into the model of insect navigation, and to implement the insect navigation mechanism on the Colias robot and test the performances and properties of the biological strategy in the real word.

In regards to the model for chip realisation, research is ongoing between those working on invertebrate visual neural system modellers and those working on the chip design. This work will develop in the second half of the project, with key partners coming together in the workshops to discuss further developments in this area.

As part of the final work package, the team propose to further combine the functionalities of LGMD1 and LGMD2 neural computations to form a hybrid visual neural network, and secondly will investigate the possibility of integrating direction and collision selective neural models with similar separated ON/OFF pathways to handle more complex visual motion challenges.

The research collaboration between partners has generated a positive impact on society in the first phase of the project. For example, the team at the University of Lincoln demonstrated the Colias robots at the University Open Days in 2017 where potential undergraduate students had been invited. The University of Newcastle, assisted by Early Stage Researchers from the University of Lincoln, organised a public lecture for a large group of primary school students on studying insects’ vision systems, using the Colias robots as demonstration tools.

In addition to the planned research activities, the consortium will also consider additional new modelling and realisation activities based on the outcomes yielded in the first phase, to exploit the potential of biologically plausible neural visual systems for various real-world applications.

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

United Kingdom

Deliverables

Other (2)

The first and second training seminars

Seminar 1: Present the early models to, and provide preliminary training on, biologically plausible neural systems for the key neural vision chip designer and researchers to apply the neural vision models to robotics. Seminar 2: Train key researchers on VLSI chip design and the identified neural system model for realization.

Kick off workshop

Contact persons and key researchers will participate to facilitate preliminary ideas exchange for all WPs.

Demonstrators, pilots, prototypes (6)

Implementation of visual neural sytems to a robotic platform

The preliminary visual neural systems will be selected and implemented individually on a robotic platform.

Neural vision chip structure identification

Investigation of optimal sensor components, techniques and tools to design the VLSI vision chip.

Preliminary demonstrator system for robust collision detection

A demonstrator system will be developed to show simple collision avoidance based on the visual-motor system.

Visual neural systems integration for single cue extraction enhancements

Integration of appropriate neural 'sub-systems' with a view to connect multiple neural systems modelling and integrate the latest experimental visual neuroscience research.

Identified visual neural systems for realization

Comparison of visual neural models.

Preliminary visual neural system models for collision cues extraction

Modelling work will take recent developments into account and continue throughout the whole project period. It will provide robust visual neural algorithms for chip design.

Documents, reports (1)

Websites, patent fillings, videos etc. (1)

Publications

Peer reviewed articles (12)

Infinite-dimensional Bayesian approach for inverse scattering problems of a fractional Helmholtz equation

Author(s): Junxiong Jia, Shigang Yue, Jigen Peng, Jinghuai Gao

Published in: Journal of Functional Analysis, 2018, ISSN 0022-1236

DOI: 10.1016/j.jfa.2018.08.002

Unidirectional Optomotor Responses and Eye Dominance in Two Species of Crabs

Author(s): Yair Barnatan, Daniel Tomsic, Julieta Sztarker

Published in: Frontiers in Physiology, Issue 10, 2019, ISSN 1664-042X

DOI: 10.3389/fphys.2019.00586

What has been missed for predicting human attention in viewing driving clips?

Author(s): Jiawei Xu, Shigang Yue, Federica Menchinelli, Kun Guo

Published in: PeerJ, Issue 5, 2017, Page(s) e2946, ISSN 2167-8359

DOI: 10.7717/peerj.2946

Shaping the collision selectivity in a looming sensitive neuron model with parallel ON and OFF pathways and spike frequency adaptation

Author(s): Qinbing Fu, Cheng Hu, Jigen Peng, Shigang Yue

Published in: Neural Networks, Issue 106, 2018, Page(s) 127-143, ISSN 0893-6080

DOI: 10.1016/j.neunet.2018.04.001

Event-Driven Continuous STDP Learning With Deep Structure for Visual Pattern Recognition

Author(s): Daqi Liu, Shigang Yue

Published in: IEEE Transactions on Cybernetics, 2018, Page(s) 1-14, ISSN 2168-2267

DOI: 10.1109/TCYB.2018.2801476

A Directionally Selective Small Target Motion Detecting Visual Neural Network in Cluttered Backgrounds

Author(s): Hongxin Wang, Jigen Peng, Shigang Yue

Published in: IEEE Transactions on Cybernetics, 2018, Page(s) 1-15, ISSN 2168-2267

DOI: 10.1109/TCYB.2018.2869384

A Robust Collision Perception Visual Neural Network With Specific Selectivity to Darker Objects

Author(s): Qinbing Fu, Cheng Hu, Jigen Peng, F. Claire Rind, Shigang Yue

Published in: IEEE Transactions on Cybernetics, 2019, Page(s) 1-15, ISSN 2168-2267

DOI: 10.1109/tcyb.2019.2946090

Fast unsupervised learning for visual pattern recognition using spike timing dependent plasticity

Author(s): Daqi Liu, Shigang Yue

Published in: Neurocomputing, Issue 249, 2017, Page(s) 212-224, ISSN 0925-2312

DOI: 10.1016/j.neucom.2017.04.003

Towards Computational Models and Applications of Insect Visual Systems for Motion Perception: A Review

Author(s): Qinbing Fu, Hongxin Wang, Cheng Hu, Shigang Yue

Published in: Artificial Life, Issue 25/3, 2019, Page(s) 263-311, ISSN 1064-5462

DOI: 10.1162/artl_a_00297

Deep Spiking Neural Network for Video-Based Disguise Face Recognition Based on Dynamic Facial Movements

Author(s): Daqi Liu, Nicola Bellotto, Shigang Yue

Published in: IEEE Transactions on Neural Networks and Learning Systems, Issue 19 July 2019, 2019, Page(s) 1-10, ISSN 2162-237X

DOI: 10.1109/tnnls.2019.2927274

A Robust Visual System for Small Target Motion Detection Against Cluttered Moving Backgrounds

Author(s): Hongxin Wang, Jigen Peng, Xuqiang Zheng, Shigang Yue

Published in: IEEE Transactions on Neural Networks and Learning Systems, 2018, Page(s) 1-15, ISSN 2162-237X

DOI: 10.1109/tnnls.2019.2910418

A 14-bit 250 MS/s IF Sampling Pipelined ADC in 180 nm CMOS Process

Author(s): Xuqiang Zheng, Zhijun Wang, Fule Li, Feng Zhao, Shigang Yue, Chun Zhang, Zhihua Wang

Published in: IEEE Transactions on Circuits and Systems I: Regular Papers, Issue 63/9, 2016, Page(s) 1381-1392, ISSN 1549-8328

DOI: 10.1109/TCSI.2016.2580703

Conference proceedings (12)

A Feedback Neural Network for Small Target Motion Detection in Cluttered Backgrounds

Author(s): Wang, Hongxin; Peng, Jigen; Yue, Shigang

Published in: Issue 1, 2018

DOI: 10.1007/978-3-030-01424-7_71

A Bio-inspired Collision Detector for Small Quadcopter

Author(s): Jiannan Zhao, Cheng Hu, Chun Zhang, Zhihua Wang, Shigang Yue

Published in: International Joint Conference on Neural Networks (IJCNN) 2018, 2018

Visual Cue Integration for Small Target Motion Detection in Natural Cluttered Backgrounds

Author(s): Wang, Hongxin; Peng, Jigen; Fu, Qinbing; Wang, Huatian; Yue, Shigang

Published in: Issue 11, 2019

Modeling Direction Selective Visual Neural Network with ON and OFF Pathways for Extracting Motion Cues from Cluttered Background

Author(s): Qinbing Fu and Shigang Yue

Published in: 2017 International Joint Conference on Neural Networks, Anchorage, Alaska, Issue 30th anniversary, 2017

Bio-inspired Collision Detector with Enhanced Selectivity for Ground Robotic Vision System

Author(s): Qinbing Fu, Shigang Yue and Cheng Hu

Published in: 2016 British Machine Vision Conference (BMVC), Issue 27th conference, 2016

Towards computational models of insect motion detectors for robot vision

Author(s): Qinbing Fu, Cheng Hu, Pengcheng Liu, Shigang Yue

Published in: 19th Towards Autonomous Robotic Systems (TAROS), 2018

A Hybrid Visual-Model Based Robot Control Strategy for Micro Ground Robots

Author(s): Cheng Hu, Qinbing Fu, Tian Liu, Shigang Yue

Published in: From Animals to Animats 15, 2018, Page(s) 162-174

DOI: 10.1007/978-3-319-97628-0_14

Generating Robust Grasps for Unknown Objects in Clutter Using Point Cloud Data

Author(s): Hongzhuo Liang, Shuang Li, Michael Görner, and Jianwei Zhang

Published in: i-CREATe 2018 Proceedings of the 12th International Convention on Rehabilitation Engineering and Assistive Technology, 2018, Page(s) Pages 298-301

Colias IV: The Affordable Micro Robot Platform with Bio-inspired Vision

Author(s): Cheng Hu, Qinbing Fu, Shigang Yue

Published in: 19th Towards Autonomous Robotic Systems (TAROS), 2018, Page(s) 197-208

DOI: 10.1007/978-3-319-96728-8_17

Path Planning for Wheeled Mobile Service Robots based on Improved Genetic Algorithm

Author(s): Shuang Li, Hongzhuo Liang, Jianwei Zhang

Published in: i-CREATe 2018 Proceedings of the 12th International Convention on Rehabilitation Engineering and Assistive Technology, 2018, Page(s) Pages 249-252

A Model for Detection of Angular Velocity of Image Motion Based on the Temporal Tuning of the Drosophila

Author(s): Huatian Wang, Jigen Peng, Paul Baxter, Chun Zhang, Zhihua Wang, Shigang Yue

Published in: 2018, Page(s) 37-46

DOI: 10.1007/978-3-030-01421-6_4

An Analysis of a Ring Attractor Model for Cue Integration

Author(s): Xuelong Sun, Michael Mangan, Shigang Yue

Published in: Living Machines 2018: Conference on Biomimetic and Biohybrid System, 2018, Page(s) 459-470

DOI: 10.1007/978-3-319-95972-6_49

Other (8)

Synthetic Neural Vision System Design for Motion Pattern Recognition in Dynamic Robot Scenes

Author(s): Fu, Qinbing; Hu, Cheng; Liu, Pengcheng; Yue, Shigang

Published in: arXiv:1904.07180 [cs.NE], Issue 1, 2019

Collision Selective Visual Neural Network Inspired by LGMD2 Neurons in Juvenile Locusts

Author(s): Fu, Qinbing; Hu, Cheng; Yue, Shigang

Published in: arXiv:1801.06452 [q-bio.NC], Issue 6, 2017

Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network

Author(s): Li, Shuang; Ma, Xiaojian; Liang, Hongzhuo; Görner, Michael; Ruppel, Philipp; Fang, Bing; Sun, Fuchun; Zhang, Jianwei

Published in: arXiv:1809.06268 [cs.RO], Issue 1, 2018

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring

Author(s): Liang, Hongzhuo; Li, Shuang; Ma, Xiaojian; Hendrich, Norman; Gerkmann, Timo; Sun, Fuchun; Zhang, Jianwei

Published in: arXiv:1903.00650 [cs.RO], Issue 1, 2019

Effects of luminance contrast on the looming-sensitive neuron of the praying mantis Tenodera aridifolia

Author(s): Suda, R and Yamawaki, Y

Published in: ELCAS Journal, Issue 4, 2019-03, 2019, Page(s) 4-10

A Resilient Image Matching Method with an Affine Invariant Feature Detector and Descriptor

Author(s): Zhao, Biao; Yue, Shigang

Published in: arXiv:1802.09623 [cs.CV], Issue 7, 2018

A Directionally Selective Neural Network with Separated ON and OFF Pathways for Translational Motion Perception in a Visually Cluttered Environment

Author(s): Fu, Qinbing; Bellotto, Nicola; Yue, Shigang

Published in: arXiv:1808.07692 [cs.NE], Issue 3, 2018

PointNetGPD: Detecting Grasp Configurations from Point Sets

Author(s): Liang, Hongzhuo; Ma, Xiaojian; Li, Shuang; Görner, Michael; Tang, Song; Fang, Bin; Sun, Fuchun; Zhang, Jianwei

Published in: arXiv:1809.06267 [cs.RO], Issue 1, 2018

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

United Kingdom

Project information

STEP2DYNA

Grant agreement ID: 691154

Project website

Status

Closed project

Start date

1 July 2016
End date

30 June 2020

Funded under:

H2020-EU.1.3.3.

Overall budget:

€ 1 228 500
EU contribution

€ 1 008 000

Coordinated by:

UNIVERSITY OF LINCOLN

Download

PDF

XML

Last update: 2 February 2020

Record number: 199944

Permalink: https://cordis.europa.eu/project/id/691154

CORDIS

Project information

Objective

Coordinator

Participants (3)

Partners (7)

Project information

Project information

Periodic Reporting for period 1 - STEP2DYNA (Spatial-temporal information processing for collision detection in dynamic environments)

Project information

Project information

Deliverables

Publications

Project information

Project information

Project information

Share this page

Download