Periodic Reporting for period 1 - Sestosenso (Physical Cognition for Intelligent Control and Safe Human-Robot Interaction)
Reporting period: 2022-10-01 to 2024-03-31
Sestosenso develops technologies for collaborative robots that self-adapt to varying conditions and transition smoothly from autonomous to interactive modes when human intervention is needed for collaboration or training. The project introduces new sensing technology from hardware to cognitive control, using networks of embedded proximity and tactile sensors on the robot body, creating a unified proxy-tactile perception for safe, autonomous interaction. These technologies are also applied to wearable devices like exoskeletons to enhance spatial awareness and safety in critical human-robot tasks.
Motivated by Industry 4.0 and Industry 5.0 principles, Sestosenso addresses the need for robots that can operate safely with minimal infrastructure, reducing setup times and costs while increasing shopfloor flexibility. The project aims to develop self-standing robot systems capable of performing in dynamic and unstructured environments without external infrastructures.
Sestosenso technology is based on networks of miniaturized proximity and tactile sensors over the robot body, providing seamless proxy-tactile perception. This technology enables:
1. Autonomous and human-driven robot operations.
2. New robot tasks using joint proxy-tactile feedback.
3. Enhanced safety in human-robot interaction (HRI).
4. Robot self-localization and environment representation.
5. New manufacturing solutions for large area distributed sensors.
Objectives of the project include demonstrating self-standing robots with proxy-tactile feedback in three industrial use cases:
1. Safe, cooperative assembly tasks in dynamic environments.
2. Outdoor tasks with unmodeled obstacles, using sensorized exoskeletons.
3. Gesture/touch guided robot teaching for handling large objects with whole-arm manipulation.
Robots will adapt to spatial changes, control contact forces, recognize human gestures, and self-localize for easy setup without external infrastructure. Proxy-tactile sensors, developed using printed electronics and additive manufacturing, aim for cost-effective, versatile designs and integration on various robots, exploring new large-area sensor concepts for reliability and cost-efficiency. Networks of real-time wireless nodes will advance cableless robot development.
Sestosenso aligns with AI, data, and robotics innovation, targeting multiple industrial and service applications and focusing on human-centered technologies.
Motivated by Industry 4.0 and Industry 5.0 principles, Sestosenso addresses the need for robots that can operate safely with minimal infrastructure, reducing setup times and costs while increasing shopfloor flexibility. The project aims to develop self-standing robot systems capable of performing in dynamic and unstructured environments without external infrastructures.
Sestosenso technology is based on networks of miniaturized proximity and tactile sensors over the robot body, providing seamless proxy-tactile perception. This technology enables:
1. Autonomous and human-driven robot operations.
2. New robot tasks using joint proxy-tactile feedback.
3. Enhanced safety in human-robot interaction (HRI).
4. Robot self-localization and environment representation.
5. New manufacturing solutions for large area distributed sensors.
Objectives of the project include demonstrating self-standing robots with proxy-tactile feedback in three industrial use cases:
1. Safe, cooperative assembly tasks in dynamic environments.
2. Outdoor tasks with unmodeled obstacles, using sensorized exoskeletons.
3. Gesture/touch guided robot teaching for handling large objects with whole-arm manipulation.
Robots will adapt to spatial changes, control contact forces, recognize human gestures, and self-localize for easy setup without external infrastructure. Proxy-tactile sensors, developed using printed electronics and additive manufacturing, aim for cost-effective, versatile designs and integration on various robots, exploring new large-area sensor concepts for reliability and cost-efficiency. Networks of real-time wireless nodes will advance cableless robot development.
Sestosenso aligns with AI, data, and robotics innovation, targeting multiple industrial and service applications and focusing on human-centered technologies.
Period 1 (Month 1 - Month 18)
During the first period, the following S/T activities were performed.
Large Area Proxy-tactile Sensors
The project developed proxy-tactile sensors embedded on the robot body. Two solutions have been studied: ProxySkin and OptoSkin. ProxySkin is planned for the final demonstrators, while OptoSkin, more innovative, is based on new optical transduction solutions.
Key achievements for ProxySkin:
• Design, implementation, and testing of printed electronic capacitive tactile transducers.
• Design, implementation, and testing of proximity sensors based on time-of-flight chips.
• Development of proxy-tactile devices for integration onto the robots for final demonstrations.
Key achievements for OptoSkin:
• Study of light-transmitting materials and sensor manufacturing solutions.
• Experimental tests of prototypes.
• Simulation and modeling to estimate contact parameters for planar and non-planar devices.
These activities led to milestones M2 and M3.
Proxy-tactile Self-awareness
These activities aim to develop active and autonomous perception and cognition of the robot's environment for HRI models and robot control algorithms. Methods for modeling objects and the environment based on sparse robot-centric data and representing proximity and tactile information were investigated.
Key achievements:
• Techniques for geometric modeling of the environment from sparse point clouds generated by ToF sensors and touch, using signed distance functions (SDF).
• Development of a multimodal middleware to provide a homogeneous representation of proxy-tactile data.
These activities led to milestone M8.
Proxy-tactile Based Gesture Recognition and HRI Control
An analysis and experimental evaluation of techniques for human presence detection, tracking, activity recognition, and motion anticipation were performed. Initial prototypes of human behavior analysis from 3D image sequences were demonstrated.
Key achievements:
• Study on behavioral models of workers interacting with robots.
• Development of a safety-aware robot planning and control framework.
These activities led to milestone M10, as reported in deliverable D5.1.
Proxy-tactile Task-oriented Control
Methods for sensor-based exploration, safe motion, and whole-arm manipulation were explored.
Key achievements:
• Study of adaptive exploration behaviors based on ergodic control and fusion of different behavior primitives.
• Study of whole-arm manipulation planning and control methods based on the SDF world representation.
• Study of a proxy-tactile reactive controller based on task-priority to minimize contacts and contact forces in cluttered environments.
Use Cases Definition, Integration, and Engineering
Sestosenso aims to demonstrate three industrially relevant use cases (UCs) for self-standing robot systems using ProxySkin sensors.
Key achievements:
• Integration of hardware, firmware, and software for the final demonstrators,
This activity is documented in D1.1 D7.1 D7.6 D7.2 D7.3 D7.4 and led to milestones M1 and M14.
During the first period, the following S/T activities were performed.
Large Area Proxy-tactile Sensors
The project developed proxy-tactile sensors embedded on the robot body. Two solutions have been studied: ProxySkin and OptoSkin. ProxySkin is planned for the final demonstrators, while OptoSkin, more innovative, is based on new optical transduction solutions.
Key achievements for ProxySkin:
• Design, implementation, and testing of printed electronic capacitive tactile transducers.
• Design, implementation, and testing of proximity sensors based on time-of-flight chips.
• Development of proxy-tactile devices for integration onto the robots for final demonstrations.
Key achievements for OptoSkin:
• Study of light-transmitting materials and sensor manufacturing solutions.
• Experimental tests of prototypes.
• Simulation and modeling to estimate contact parameters for planar and non-planar devices.
These activities led to milestones M2 and M3.
Proxy-tactile Self-awareness
These activities aim to develop active and autonomous perception and cognition of the robot's environment for HRI models and robot control algorithms. Methods for modeling objects and the environment based on sparse robot-centric data and representing proximity and tactile information were investigated.
Key achievements:
• Techniques for geometric modeling of the environment from sparse point clouds generated by ToF sensors and touch, using signed distance functions (SDF).
• Development of a multimodal middleware to provide a homogeneous representation of proxy-tactile data.
These activities led to milestone M8.
Proxy-tactile Based Gesture Recognition and HRI Control
An analysis and experimental evaluation of techniques for human presence detection, tracking, activity recognition, and motion anticipation were performed. Initial prototypes of human behavior analysis from 3D image sequences were demonstrated.
Key achievements:
• Study on behavioral models of workers interacting with robots.
• Development of a safety-aware robot planning and control framework.
These activities led to milestone M10, as reported in deliverable D5.1.
Proxy-tactile Task-oriented Control
Methods for sensor-based exploration, safe motion, and whole-arm manipulation were explored.
Key achievements:
• Study of adaptive exploration behaviors based on ergodic control and fusion of different behavior primitives.
• Study of whole-arm manipulation planning and control methods based on the SDF world representation.
• Study of a proxy-tactile reactive controller based on task-priority to minimize contacts and contact forces in cluttered environments.
Use Cases Definition, Integration, and Engineering
Sestosenso aims to demonstrate three industrially relevant use cases (UCs) for self-standing robot systems using ProxySkin sensors.
Key achievements:
• Integration of hardware, firmware, and software for the final demonstrators,
This activity is documented in D1.1 D7.1 D7.6 D7.2 D7.3 D7.4 and led to milestones M1 and M14.
The scientific achievements beyond the state of the art have been published on: 9 peer reviewed journal papers, 3 peer reviewed conference papers, and 5 papers currently under revision (Aprile 2024).
A major result for the future exploitation of the Sestosenso technologies is a patent on the OptoSkin technology.
Details about this dissemination and exploitation activities can be found on the project's website: www.sestosenso.eu.
Periodic updates about publications and general project activities are published on
- X: twitter.com/sestosenso_eu
- Linkedin: https://www.linkedin.com/company/sestosenso-eu/(opens in new window)
A major result for the future exploitation of the Sestosenso technologies is a patent on the OptoSkin technology.
Details about this dissemination and exploitation activities can be found on the project's website: www.sestosenso.eu.
Periodic updates about publications and general project activities are published on
- X: twitter.com/sestosenso_eu
- Linkedin: https://www.linkedin.com/company/sestosenso-eu/(opens in new window)