Periodic Reporting for period 1 - MAGICIAN (iMmersive leArninG for ImperfeCtion detectIon and repAir through human-robot interactioN)
Reporting period: 2023-10-01 to 2025-03-31
MAGICIAN will develop AI-driven robotic solutions for the autonomous identification and repair of aesthetic defects in manufacturing products. As consumers increasingly expect manufacturing products to be free of defects, production processes follow high standards. In addition, tasks associated with production processes in manufacturing (e.g. grinding) are physically and cognitively demanding for workers and executed in a potentially hazardous environment. The robotic solutions developed in the project will rely on data collected from human workers’ operations and aim at:
- making production processes more efficient, and
- improving safety conditions of human workers by shifting physically demanding and hazardous tasks to the robots while human workers will perform tasks with a more cognitive focus (e.g. supervision and control)
The robotic solutions will be first tested and piloted in an automotive use case and then expanded to additional use cases and application fields through two Open Calls for Start-ups and SMEs.
Overall, the project follows a human-centred approach, allowing for a sustainable transition towards human-robot collaboration in manufacturing based on trust and safety.
- making production processes more efficient, and
- improving safety conditions of human workers by shifting physically demanding and hazardous tasks to the robots while human workers will perform tasks with a more cognitive focus (e.g. supervision and control)
The robotic solutions will be first tested and piloted in an automotive use case and then expanded to additional use cases and application fields through two Open Calls for Start-ups and SMEs.
Overall, the project follows a human-centred approach, allowing for a sustainable transition towards human-robot collaboration in manufacturing based on trust and safety.
Comprehensive Manufacturing System Analysis: A detailed description of the production systems, including machines, robots, their functions, and connection types, was developed based on on-site assessments at TOFAS in Bursa, Turkey (January 2024). The roles of human operators in collaborative tasks were identified and analyzed to inform Human-Robot Collaboration system design. SSH considerations were embedded to ensure meaningful work, ethical integration, and holistic system design—balancing social and technical factors to protect human well-being. The analysis also captures the Use Cases from a business-driven user perspective, detailing project objectives, operational priorities, and the KPIs for evaluating impact and success.
Baseline studies: After the ethical approvals obtained for user-related studies in both Sweden and Turkey, interviews with workers, managers and developers have been conducted in TOFAS (Turkey) and Volvo Cars (Sweden). A comprehensive documentation created on the human-centred activities carried out has been analysed and the collected material submitted in a journal paper. Moreover, development of workshop materials is in progress, targeting diverse stakeholder engagement for design activities.
Integrated Multi-Modal Perception System: It has been developed a hardware/software prototype system combining vision- and tactile-based sensing to replicate human defect detection, with data acquisition for training classification models. The perception modules are integrated with the CAD models and metrological constraints to support precise, context-sensitive detection. A set of learning-based methods for defect classification using multi-modal sensory input, mimicking expert operators, has been developed, thus laying the foundation for integrating the learned models into robotic control systems.
Human-Aware Planning: Advanced human motion prediction used to inform downstream human-aware planning to set the stage for context-adaptive behaviour in collaborative scenarios. The human motion prediction models are fed by improved motion capture techniques to ensure accurate modelling of human reworking motions, also used for transfer learning algorithms for machine learning of human skills, ready for robotic system integration.
Human-Robot Interfaces with Tactile Perception: intuitive human-robot interfaces integrating tactile perception for improved surface defect detection have been designed, which supports learning-by-demonstration.
Design of Robot-Grinder Interface: a comprehensive analysis of robot-grinder interactions has been conducted, including vibrational dynamics and contact forces. The result is an optimized end-effector design and a set of adaptive control algorithms to handle varying defect types and operational conditions.
Efficient Inspection, Reworking Task Planning and Motion Control: a motion planning framework using an ad-hoc Probabilistic Road Map (PRM) approach has been conceived for the MAGICIAN reworking operation. The planner is executed offline and real-time modules for optimising the task sequences, balancing time, energy, and efficiency have been grounded onto the orienteering based approach, factoring in defect severity, type, and location. A motion controller, based on ergodic control algorithms for both visual and tactile active sensing, is used to execute plans with a focus on safety, ergonomics, and efficiency has been developed on purpose and follows the indication collected during the TOFAS site visit.
Spatter Detection and Optimisation Tool: The spatter detection requirements has been gathered via interviews with workers and quality managers alongside with the collection of real welding data to initiate the training of statistical and physical models for resistance spot welding to detect the spatter from welding gun measurements. To this aim, the focus is on feature extraction algorithms to improve detection speed.
Baseline studies: After the ethical approvals obtained for user-related studies in both Sweden and Turkey, interviews with workers, managers and developers have been conducted in TOFAS (Turkey) and Volvo Cars (Sweden). A comprehensive documentation created on the human-centred activities carried out has been analysed and the collected material submitted in a journal paper. Moreover, development of workshop materials is in progress, targeting diverse stakeholder engagement for design activities.
Integrated Multi-Modal Perception System: It has been developed a hardware/software prototype system combining vision- and tactile-based sensing to replicate human defect detection, with data acquisition for training classification models. The perception modules are integrated with the CAD models and metrological constraints to support precise, context-sensitive detection. A set of learning-based methods for defect classification using multi-modal sensory input, mimicking expert operators, has been developed, thus laying the foundation for integrating the learned models into robotic control systems.
Human-Aware Planning: Advanced human motion prediction used to inform downstream human-aware planning to set the stage for context-adaptive behaviour in collaborative scenarios. The human motion prediction models are fed by improved motion capture techniques to ensure accurate modelling of human reworking motions, also used for transfer learning algorithms for machine learning of human skills, ready for robotic system integration.
Human-Robot Interfaces with Tactile Perception: intuitive human-robot interfaces integrating tactile perception for improved surface defect detection have been designed, which supports learning-by-demonstration.
Design of Robot-Grinder Interface: a comprehensive analysis of robot-grinder interactions has been conducted, including vibrational dynamics and contact forces. The result is an optimized end-effector design and a set of adaptive control algorithms to handle varying defect types and operational conditions.
Efficient Inspection, Reworking Task Planning and Motion Control: a motion planning framework using an ad-hoc Probabilistic Road Map (PRM) approach has been conceived for the MAGICIAN reworking operation. The planner is executed offline and real-time modules for optimising the task sequences, balancing time, energy, and efficiency have been grounded onto the orienteering based approach, factoring in defect severity, type, and location. A motion controller, based on ergodic control algorithms for both visual and tactile active sensing, is used to execute plans with a focus on safety, ergonomics, and efficiency has been developed on purpose and follows the indication collected during the TOFAS site visit.
Spatter Detection and Optimisation Tool: The spatter detection requirements has been gathered via interviews with workers and quality managers alongside with the collection of real welding data to initiate the training of statistical and physical models for resistance spot welding to detect the spatter from welding gun measurements. To this aim, the focus is on feature extraction algorithms to improve detection speed.
The projects aims at improving the applicability, efficiency and acceptability of AI-powered robotic solutions in manufacturing. The direct impact of the developed solution will be in automated defect detection and reworking, but the potential impact goes beyond the specific application scenario. To this end, the project is aiming to produce impactful relevant solutions and publications, with a special focus on production processes, which demand further research, investigation and the first validation in the demonstrator foreseen in the rest of the project.
The ambition of MAGICIAN is also to provide regulatory and standardisation results, first for the automotive domain and then, with the second round of OCs, for other application domains.
The ambition of MAGICIAN is also to provide regulatory and standardisation results, first for the automotive domain and then, with the second round of OCs, for other application domains.