Community Research and Development Information Service - CORDIS


SatisFactory Report Summary

Project ID: 636302
Funded under: H2020-EU.

Periodic Reporting for period 1 - SatisFactory (A collaborative and augmented-enabled ecosystem for increasing SATISfaction and working experience in smart FACTORY environments)

Reporting period: 2015-01-01 to 2016-06-30

Summary of the context and overall objectives of the project

Transfiguration of traditional industrial environments into attractive and safe workplaces is a key to the Industrial Revolution. SatisFactory (“A Collaborative and Augmented-Enabled Ecosystem for Increasing Satisfaction and Working Experience in Smart Factory Environments”) is a three-year long research project funded by the European Commission under the HORIZON2020 programme, which started in January 2015 to tackle this problem.

Manufacturing is vital component of our society, but can only realise its full potential if it can embrace the ongoing changes in global economy and technology. It should be adapted on the new opportunities that are stemming from the development and integration of “factories of things”. Some of the key issues that should be addressed are: the enhancement of industrial infrastructure, the incorporation of intelligent technologies in manufacturing environments, the development of intelligent human-centric technologies but more importantly the assessment and communication of the employee’s expertise, through the stimulation of team interactions. SatisFactory envisions a factory of the future as a healthy place providing pleasant working experience. It is a place where staff members feel appreciated and valued for their contribution and are delighted to come to work every day. This enhancement of the overall working experience in a factory in combination with appropriate marketing and recruiting strategies will make industrial employment more attractive to potential younger applicants as well as enhance the wellbeing and satisfaction of the employees.

SatisFactory aims to enhance and enrich the manufacturing working environment towards attractive factories of the future that encompass key enabling technologies such as augmented reality, wearable and ubiquitous computing as well as and customised social communication platforms coupled with experience design and gamification techniques for the efficient transfer of knowledge and experience among employees. The ultimate goal will be to develop and deploy emerging knowledge-driven training techniques (gamification aware) and wearable devices (AR-enabled Glasses) for the enhancement of innovation, productivity and scheduling of work in the production line while enriching its flexibility through the support of team interactions in the shop floor which consequently will add to the work-related satisfaction. Special attention in the deployment of cutting-edge incentives (wearable computing, gamified working experience) will be given towards further increase of attractiveness to the younger generation. The focus on user experience and gamification will ensure optimal balance between workers performance and satisfaction.
To this end, the SatisFactory framework will be comprised by five major components: (a) The Smart Sensor Network that consists of the heterogeneous devices that will allow the interaction between the physical world and SatisFactory framework. This network will be integrated through Middleware. (b) The Ontology Manager which processes the data collected by the Smart Sensor Network and transform them into semantically enriched events. (c) The integrated Decision Support System which processes the events and semantically enriched data, and is responsible for supporting decision making in manufacturing processes and operations. (d) The Augmented Reality In-Factory platform, which provides additional relevant data to the employee while working, or to reach a realistic experience while being trained in a simulated framework. (e) The training/educational environment, which consists of a set of components that are used for in-factory training and support of the workers. AR technologies are used to deliver “on-the-job” training and education for not only workers and machinery operators but also for manufacturing process supervisors.

In order for the SatisFactory project to successfully reach its goals, several prerequisites are set in the form of major Scientific and Technological Objectives throughout the duration of the project:
Objective 1: Context-aware control and re-adaptation of shop floor production facilities for increased productivity and flexibility in use of shop floor resources. To develop a lightweight middleware for interconnecting the SatisFactory components. A semantically enriched framework able to retrieve the historic and real-time data from the shop-floor extracting semantically enhanced information, especially regarding the HR activities should be developed, in order to support the HR re-adaptation toolkit. Finally, functionalities and methods for the HR re-adaptation will be investigated and developed, which will be able to assign the daily tasks to the employees, and to make optimal re-assignments when a new un-scheduled task appeared, increasing the productivity and the flexibility of the shop-floor resources.

Objective 2: Improvement of attractiveness and productivity through collaboration, social interaction and gamification approaches. To develop a web-based collaboration platform, social interaction and gamification framework supporting a variety of functionalities such as users, groups, posts, timeline indicators, multimedia exchange, gamification points and awards, etc. These approaches should be designed and implemented in order to increase the interesting of the employees for the factories productivity.

Objective 3: Real-time knowledge-sharing and AR-based collaboration and training services. To develop real-time knowledge sharing mechanisms providing the ability to the workers exchanging useful information regarding the daily tasks. AR technologies for collaboration and training services should be designed and developed increasing the interactivity between the employees via an attractive environment.

Objective 4: Improved shop floor feedback and decision making for gains in productivity, workers wellbeing and comfort. Novel algorithms for the collection and analysis of information received by the Smart Sensor Network installed in the shop-floor will be research and developed, providing information regarding potential incidents in the shop-floor and their on-time detection and recognition. Furthermore, a decision support system will be designed and implemented for making optimal decisions concerning the raw information from the sensorial network, the events (e.g. human falls, collisions, thermal incidents, etc.) produced by the SatisFactory components and the semantically enriched events provided by the Ontology Manager.

Objective 5: Adaptive and augmented interfaces for collaboration, knowledge sharing and real time support. Novel wearable HMIs (e.g. AR glasses, wearable devices for localization, etc.) should be designed and developed providing through a variety of adaptive and augmented interfaces, functionalities for real-time collaboration and knowledge sharing between the workers.

Objective 6: Deployment and Evaluation of the solution to large-scale Industrial Facilities from the Automotive and Energy factory domains. To this end, deployment and evaluation of the SatisFactory framework will take place in three pilot shop-floors (CERTH/CPERI, COMAU and SUNLIGHT), in Greece and in Italy.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the first period of the SatisFactory project, the SatisFactory Consortium has performed significant progress strictly following the project’s Description of Action (DoA –Annex I). The work in the first period of the project (M1-M18) focused on the user requirements analysis and use cases definition, as well as on the overall SatisFactory system design and specification. In parallel, the development efforts have been proceeded in all domains of the SatisFactory project, e.g. collaboration platform, gamification framework, HR re-adaptation toolkit, context-aware managers, decision support engine, IoT infrastructure, AR glasses, AR toolkit and “on-the-job” education toolkit, along with the technical developments of the SatisFactory hardware infrastructure. Moreover, project efforts focused also on several horizontal activities regarding diffusing project objectives, concepts and achievements in key stakeholders as well as to the general public.

Very briefly, the developments for the reporting period can be summarized but are not limited to the following points:

- Definition of end-user, shop-floor and system requirements and specifications
The collection of end-user needs and system requirements and specifications is the fundamental basis for the development of any system or product. Furthermore, the accurate understanding of user requirements is specifically important for the introduction of the solutions in the shop-floor, since they address among others safety and ergonomic issues, training, etc. In this direction, the definition of the requirements has been performed with the support of specific material, like questionnaires and by performing face-to-face meeting and workshops in order to apply as much as possible the concept of the user-centered design. The SatisFactory project involved the participation of 3 shop-floor from 2 different countries (Greece and Italy) and 41 users (18 from CERTH/CPERI, 15 from COMAU, and 8 from SUNLIGHT). All the collected material has been analysed and the most relevant concepts have been extracted and summarized in requirements and specifications. This process is iterative and it is actually in its second iteration out of three. Finally, 89 requirements (functional & non-functional) have been defined so far.

Related Submitted Deliverables: D1.1 “User group definitions, end-user needs, requirement analysis and development guidelines”.

- Actors and procedures interconnection modelling
The core scope of the actor & procedures modelling framework is to identify and create the necessary models for the high-level analysis of the business processes of the industrial domains introduced by Satisfactory project. At a first towards fulfilling this objective, a detailed analysis has been performed for identifying and analyzing existing standards and available tools for eventually selecting the most appropriate modelling framework to be applied in Satisfactory. The performed activities covered the full Business Process Modelling lifecycle (BPM) analysis in order to focus on the modelling standards that can be used in the project for capturing the industrial standard operating procedures. Efforts in this respect concluded with the 1) classification of the procedures (e.g. manufacturing, maintenance, training, etc.) and the 2) identification of specific needs and requirements related to the management of procedures.
Based upon the findings of the survey conducted, a concrete methodology (5-steps) based on multi-actor and multi-criteria approach has been derived for analyzing the actors & procedures of the Satisfactory shop floors. In this context, several groups of actors have been created taking into consideration the role, the level of experience, the knowledge and the responsibilities of each actor. Moreover, detailed interviews with the actors at each shop floor were conducted in order to better analyze the shop floors of Satisfactory. Furthermore, in order to have a unified approach, to keep track of all the steps to be taken, to avoid gaps, to better understand and to keep record of all the procedures there was an effort to formalise them and to create a common template that would describe their functionalities. The general concept was to unify and simplify the description of the procedures at a high level. So, it was necessary to break down the procedures’ description into different use cases in order to be studied in details and to be categorised as needed. Finally, three main categories of actors have been defined, which were splitted to eleven (11) subcategories, while the procedures are directly related to the use-cases and the defined business scenarios.

Related Submitted Deliverables: D1.2 “Use Case Analysis and Application Scenarios Description”.

- Definition of the SatisFactory Use-Cases and Application Scenarios
In order to guarantee a proper assessment and validation of the Satisfactory solutions as well as to provide a number of important targeted indicators of framework usability and functionality, the collected user and system requirements were combined for the definition of eleven business scenarios and twenty one cases (UCs). These scenarios have been correlated with Satisfactory objectives in order to ensure that every concept introduced is covered by more than one application scenario and ensuring that a multitude of options are available to demonstrate the potential and validate the effect of the toolkits that will be integrated in each shop floor. Overall, the application scenarios defined cover dynamic operations and activities related to 1) Real-time support activities, 2) Event and incident logging, 3) Learning and Collaboration environment and have been selected to better highlight the user & system requirements, targeting to address existing issues or barriers related to the knowledge sharing and collaboration of actors within shop floor. In total six high-level categories of use cases have been identified and for each of them, detailed sub-use cases have been drafted for describing in the details the core functionalities anticipated by the Satisfactory platform, fully presenting the interactions among identified actor groups and architectural components of the Satisfactory framework.

Related Submitted Deliverables: D1.2 “Use Case Analysis and Application Scenarios Description”.

- Common Information Data Exchange Model (CIDEM)
The SatisFactory Common Information Data Exchange Model (CIDEM) defines the high level domain model comprising the basic elements (events, relations, interfaces etc.) underlying the SatisFactory ecosystem.
CIDEM elements were defined (XSD schemas), which are the basis for the data exchange between SatisFactory components. Also, CIDEM constitutes a repository that hosts shared and common knowledge vocabulary, through which it enables addressing the information needs of the SatisFactory project. The access of the data is performed through CIDEM API, which is defined for all CIDEM elements. Currently, CIDEM can support multiple shop-floors, and all the information needed from the SatisFactory components.
Although the CIDEM has been completed both in terms of schemas and services, it is considered as an ongoing component since occasional minor modifications are needed to optimise the alignment with the SatisFactory components.

Related Submitted Deliverables: D1.3 “SatisFactory Common Information Data Exchange Model”.

- Definition of the guidelines for deployment
During the design phase of systems, it is often assumed that some required infrastructure exists or will exist. Deploying the designed systems is the precondition for using them and then this infrastructure is often not as assumed. The consequences are disagreement and rejection of technology, accidents, incidents or work inefficiency. Improvements in technology are often less important than barriers to deployment. Hence, we put a particular focus to study requirements for deployment. This is meant as input to what all work packages should keep in mind.
To prove the need for specific requirements related to deployment, a literature research was performed, during which several papers were identified. To identify the concrete requirements, a combination of observation, interviews, literature research, and template-based requirement engineering were used. The Satisfactory use cases were analysed in a holistic manner (process, actors, and systems introduced by Satisfactory). It was thus made possible to identify the hurdles and provide a view on the challenges and possible solutions for successful deployment.
Literature research helped save time and identify concrete problems from other researchers’ experience. Additionally, it provided a viewpoint external to the consortium itself. For instance, problems are guaranteed to happen, and the installation will often be done by several people, working at times after each other, at times without even communicating to each other.
Several interviews at SatisFactory business partners were carried out, with employees both in the shop floor and at managerial levels. The aim was to uncover daily annoyances with the workflow, as well as experience and problems encountered during installation of other factory-related systems. Observations of their work were also performed, noting the tools they carried and their work processes and how they all intertwine with each other.
Finally, the Volere template was used as a checklist containing all possible requirement types, seeing what was applicable in a deployment scenario. It should be noted that the Volere template was not the primary tool used, but rather a checklist to make sure that no requirements have been forgotten. Where such cases were found, the requirements that the Volere template suggests were discussed within the group of usability experts from Fraunhofer FIT, to make sure that these requirements are indeed relevant to Satisfactory.
Currently 11 deployment requirements have been defined so far.

Related Submitted Deliverables: D1.1 “User group definitions, end-user needs, requirement analysis and development guidelines”.

- Definition of the SatisFactory architecture
The architecture of a software system is the fundamental basis upon which all design and development processes should be based. In this direction the work of the SatisFactory in the first reporting period included the definition of the first and revised version of the system’s architecture. More specifically, the different viewpoints (e.g. functional, deployment, information views) of the architecture were described and the methodology of their definition was outlined. As the following step the conceptual architecture of SatisFactory, as it was presented in the DoA of the project, was enhanced and adapted to the identified user and technical requirements. The current (second) version of the architecture has been separated into five main layers: the Physical Layer, Decision Layer, Facility Layer, Service Layer and Attractive User Interface (UI) Layer. The integration of all layers is managed by the Linksmart Middleware module, which handles the heterogeneity of physical devices and allows messages and events to be exchanged among the SatisFactory components and services. The Middleware includes three main subcomponents, namely: Device Manager, Event Aggregator and Event Manager. For each layer a summary of the functional modules has been drafted along with the interactions with other layers and/or components, whereas the anticipated internal and external interfaces (inputs and outputs) were defined in details. Standard diagrams (UML) were created for indicative use-cases illustrating the interconnection and the communication of the SatisFactory components.

Related Submitted Deliverables: D2.1 “SatisFactory System Architecture”.

- Semantically enriched framework for dynamic analysis of shop-floor operations
The development of a semantically enriched framework for analysis of shop floor operation, with the use of state of the art tools for semantic interoperability, aims at of capturing knowledge on dynamically evolving shop floor operations. Knowledge can be represented through the use of semantic models (ontologies) together with rules for knowledge extraction and inference. The Ontology Manager (OM) architecture documented on D2.2 shows the hierarchical structure of the latter, which leverages the strengths of a semantic model at different levels of abstraction, from the data-oriented ontology to the domain-specific (shop floor-oriented) ontologies.
The software stack (Open Semantic Framework) was used to deploy the Ontology Manager consists of multiple layers. The framework provides a service-oriented architecture, fully aligned with the Satisfactory overall framework specifications.
Furthermore, the analysis of the shop floor information aims the extraction of the SatisFactory knowledge, which can be (re)used to perform enriched assessments. In this framework, a set of use cases have been implemented to show the exploitation aspects of the knowledge framework.

Related Submitted Deliverables: D2.2 “Knowledge model for human resource optimization”.

Related Publication: A. Tsolakis, D. Arena, S. Krinidis, A. Perdikakis, D. Ioannidis, D. Kyritsis and D. Tzovaras, “Semantically Enriched Industry Data & Information Modelling: A feasibility study on Shop-floor Incident Recognition”. 14th IEEE International Conference on Industrial Informatics (INDIN’16), Futurescope, Poitiers, France, 18-21 July 2016

- Novel tools for increasing attractiveness of the workplaces
a) Social Collaboration Platform: One of the core aims of the project is to integrate novel concepts to the factory processes that will enhance productivity through increased engagement of workers in the processes and through the stimulation of collaboration. User experience and gamification approaches will improve the commitment of workers to less popular concepts and at the same time will increase the attractiveness of the shop floor environment for younger workers. This effect will be strengthened by the introduction of a social collaboration platform in which workers may exchange work knowledge / experiences / practices, but also social and entertaining information when appropriate. Towards this direction and taking into account the End-User requirements drafted early in the project lifetime, a first version of the collaboration platform has been delivered, introducing techniques for interpersonal communication, problem solving, continuous learning and teamwork. The platform currently enables co-workers of the shop floor to communicate and assist each other by forming online social relationships and posting multimedia content related to their every shop floor activities. The social engine enables a worker to be connected with another one or follow other workers, allowing him/her to be informed of their activities, exchange messages, etc. Workers can post text, images and videos on their timelines and perform the usual social actions such as leaving comments, sharing and liking the posted content. They may also post questions and answers in a Q&A forum, they can use the envisioned suggestions platform, exchange online messages in order to solve problems rising from the dynamic evolving factory processes, view the incidents occurring on the shop floor, etc. The first version of the platform has been installed at the pre-industrial Trial in CERTH/CPERI premises, in which feedback received from the workers at the shop floor will be used for the delivery of the enhanced version. The final (enhanced) version will be installed in the two industrial environments (COMAU & SUNLIGHT) for evaluation by respective end-users.

b) Gamification platform: Another important component for improving the attractiveness of industrial workplaces is the gamification platform. Ga
mification is defined as the use of conceptual elements that are typical for games, in a non-game context. Mostly, the aim is to motivate people, e.g. for less challenging, monotonous, or too complex tasks. As first step, the existing gamification elements in literature were surveyed. In the following, the gamification framework, which is built for SatisFactory, has been drafted based upon user and system requirements. In total, more than ten different gamification elements have been defined and implemented. The overall aim was to create a holistic gamification environment at a factory, which is not restricted to single actions performed by individuals or teams at the shopfloor. The gamification framework provides a simple to use restful API, which can be used by the other components to create games and to notify the gamification framework for the achievements of the groups or individuals. Currently, six (6) different SatisFactory components are connected to the gamification framework. They consist of 15 different gamified actions. In addition there is the connection to the digital andon and the social collaboration platform as visualization technologies and to the CIDEM for storage. The concrete games will be described in Deliverable D2.3.

c) Suggestions for improvements platform: The suggestions for improvement platform is a good example of how the user-centered design process leads to systems which arise from a concrete analysis of the specific needs of the investigated user group. During the iterative requirements engineering process, it was detected that workers in the SatisFactory pilot factories appreciate the possibility to provide suggestions for improvement a lot. So, an electronic “Suggestions for improvement” system has been designed and implemented.
To this end, a GUI has been designed and implemented. Each suggestion can be upvoted and downvoted. Screens for two tablet apps were implemented: The MyFactory App and the MyFactoryManagement App. The first is for the workers, the second is for the decision-makers. Both can also be accessed from desktop browsers. MyFactory is meant to be used as a kiosk, a fixed tablet at the shop-floor which every worker can access. MyFactoryManagement shall be used from the office PCs of decision makers. The platform is connected to the gamification framework. Submitting a suggestion, getting a suggestion accepted and voting are the gamified actions implemented so far.

d) Digital Andon: It is a component that represents a Novel HMI and is meant to be deployed at the shop floor level. It acts as a content management system and allows multilayer data visualization. This component adds the advance functionality of displaying dynamic information, creating multiple layers and rendering them over heterogeneous canvases. Each layer is provided with its own regions and resources in order to increase the dynamic nature of the component itself, allowing third-party in-factory components to draw on every SatisFactory enabled display in a more comprehensive manner.

The first version of the tools developed for improving industrial environments attractiveness and factory process productivity will be documented in Deliverable D2.3 (M20).

- Human resources optimization and re-adaptation toolkit
Satisfactory aims to optimally use the aggregated knowledge of the shop floors in order to leverage the control and re-adaptation of facilities, including the optimization of human resources and the underlying everyday worker’s workload efficiency and schedules. Towards this direction, several modules comprising the HR optimization and re-adaptation framework has been delivered: 1) The Work Schedule Prioritisation Module (WSPM), which is responsible for creating automatically the daily schedule, 2) The re-adaptation module that analyses the scheduled & unforeseen tasks in the shop floor and provides HR workload management & optimization. Multiple criteria are taken into account for the selection of the most suitable human resource such as the workload already assigned, the priority of the new task, the expertise required, and other. Moreover, additional information about the suitability of employees for a specific task is utilized, based on historical information (task allocation data and other criterial related to suitability such as ratings provided by the semantically-enriched framework). Different types of tasks are supported: atomic tasks, which require one employee only, and composite tasks which are composed of sub-tasks and involve employees of different trades. When a new task arrives, the scheduling solution of the minimum cost is selected. The goal is (1) to schedule a new urgent task as soon as possible, (2) assign suitable human resources to the task, (3) to reduce the disturbance to the work schedule caused by the addition of the new task and (4) to keep the assigned workload balanced among employees. The HR re-adaptation component retrieves static information about employees and tasks from CIDEM by utilizing its API. The component can be regarded as part of the Satisfactory DSS, whereas it is accompanied by a web-based interaction framework for enabling supervisors (shop floor managers) to automatically receive notifications of specific events on the shop floor, schedule works and watch running machinery in a single view. The main views that are currently supported by the toolkit is 1) the map view, 2) the work-schedule view and the 3) incident replay view. The map view allows the supervisor to have an immediate view of the shop floor as a whole with both the machinery and the workers. The work schedule view provides an intuitive visualization of the past, current and future tasks, while the incident replay view allows watching and re-playing past video recordings of some events considered significant and previously stored in the system.

Related Publication: S. Zikos, S. Rogotis, S. Krinidis, D. Ioannidis and D. Tzovaras, “Human-Resources Optimization & Re-Adaptation Modelling in Enterprises”, 11th European Conference on Product & Process Modelling (ECPPM’16), Limassol, Cyprus, 07-09 September 2016

- Context-aware components to enable Knowledge Share & support Decision Making process
a) Localization manager (LM): To increase factory productivity as well as safety it is very important to analyze the current position of workers, in particular in workplaces where incidents can occur and localisation of alerts is vital for workers safety. The LM is a software module that provides safety related alerts based on a geo-fencing logics service. In particular, given a set of predefined forbidden areas, the LM detects if workers are inside these areas on the basis of worker positions estimated by the UWB wearable devices. The overall algorithm and the corresponding hardware devices (e.g. UWB wearable devices) have been designed and implemented during the reporting period. When an incident is detected, an event is generated, which is sent through Middleware to all SatisFactory components (mainly to the DSS) that are interested in for further analysis. Furthermore, a simple GUI for the localization application has been developed for testing and demonstration purposes.
b) Multiple-media manager: An important component of the Context-Aware Manager is the Multiple-Media Manager. It is a component that acts as a proxy to provide distribution of the contents generated by media sources, e.g. the Gesture & Content Recognition Manager (GCRM). This kind of components in fact can process the data produced by heterogeneous devices (both video and metadata), but do not have built-in streaming functionalities. The Multiple-Media Manager allows the distribution of both visible and infrared video information in deferred mode. A deferred video is a stream provided when an incident occurs and it provides the recording of the last minutes before the incident, in order to inform supervisors about the event with detailed visual information. The Multiple-Media Manager has been designed and implemented, while the integration with the other SatisFactory components has been initiated.
c) Gesture and content recognition manager (GCRM): The Gesture & Content Recognition Manager (GCRM) is a service dedicated to the analysis of video streams (both RGB and depth) at the level of the Smart Assembly Station (SAS). It provides high level information that deliver advanced services (e.g. presence detection and people count, safety gear detection, fall detection, gestures detection) for an effective support of workers in the shop-floor. The proposed GCRM module has been explicitly designed and implemented to live in the Smart Station area based on privacy preserving technology.
d) Incident detection algorithms: SatisFactory will support workers’ wellbeing and comfort by providing incident detection capability on the shopfloor. Towards fulfilling this objective, the following components have been designed and developed:
i) Incident detection based on depth cameras: A real time, multi-camera, multi-space system has been designed and implemented for incident detection utilizing depth cameras/sensors. The system recognizes incidents such as falls, intrusion to restricted areas, collisions and falling items and, according to the type of incident and triggers the appropriate alarm in order to notify the SatisFactory components. The toolkit is an important toolkit for the increase of the workers’ safety. The developed incident detection method comprises three steps: 1) detection and tracking of moving items in the monitoring area, 2) extraction of event features, i.e. features that indicate the tracked item’s state such as its velocity, area or distance from a region of interest, 3) an HMM method that models and recognizes the occurred incident based on the values of the event features.
ii) Incident detection based on thermal cameras: During the period covered in this report, a set of image processing and computer vision algorithms that aim to identify overheated areas, have been designed and developed. These regions of interest significantly alter the thermal signature of the inspected equipment and generate appropriate alarms. The proposed tools come to address identified needs described in BSC-4.2 “Monitoring of the Cell Temperature during Jar formation and data collection” and in BSC-6.1 “Recognition of incidents and path optimization for workers movement on the shop floor”. It is of utmost importance, since in both cases can save time from the workers, while on the same time it can increase their safety in the shop-floor.
A problem that makes thermal cameras not widely used in the incident detection tools, is their difficulty to transform their local coordination system to a global one. Within SatisFactory project a novel thermal camera calibration procedure has been designed and implemented that attempts to increase situational awareness, support proactive/preventive maintenance and thus lead to safer worker environment. To this end, a calibration pattern for thermal cameras was manufactured and prepared exhibiting high thermal contrast. Consequently, a geometric camera calibration is performed andand then linked against building information, stored within BIM files, in order to generate a detailed 3D thermal model of the inspected area.

- SatisFactory decision support system (DSS)
a) Core DSS engine: One of the most important components of the SatisFactory solution is the DSS engine, which is responsible for providing feedback to the decision makers about shop floor incidents needing immediate response or action delegation. It also suggests the changes to processes and maintenance operations, as well as schedules. To this end, a complicate DSS engine has been designed and implemented, which receives input from the shop-floor (e.g. existing automation system, the Smart Sensor Network, Ontology Manager, Incident Detection components, etc.) and makes suggestions on how to react to each incident, to re-adaptation processes, re-scheduling activities, etc. through triggering actions in other SatisFactory component (especially visualization and alert components).
b) Signal processing algorithms & Shopfloor Data analytics
i) Incident detection - Real-Time Slope Statistic Profile (RTSSP) method: An important component of the SatisFactory solution is the real-time incident detection occurred during the factory processes execution (e.g. the chemical reactors in CERTH/CPERI pilot). To this end, a Real-Time Slope Statistic Profile (RTSSP) algorithm has been designed and implemented for the early malfunction diagnosis in real time. The algorithm receives all the respective data from the shopfloor (e.g. reactor at CPERI) and estimates the change point of the linear trend in a time series analysis, from its linear trend test statistic, computed on consecutive overlapping sliding data window along the time series. The algorithm is adaptive and supports dynamic changes of the sliding window length improving both accuracy and performance.

Related Publication: T. Vafeiadis, S. Krinidis, C. Ziogou, D. Ioannidis, S. Voutetakis and D. Tzovaras, “Robust malfunction diagnosis in process industry time series”, 14th IEEE International Conference on Industrial Informatics (INDIN’16), Futurescope, Poitiers, France, 18-21 July 2016

ii) Incident detection - Multi-Correlation Index (MCI): MCI is another real-time incident detection algorithm with ultimate goal the identification of situations/ modes in the behaviour of a factory process (e.g. thermocouples in a chemical reactor of CPERI). There are three modes that need identification: a) normal mode, where all monitored units (i.e. thermocouples) are functional, b) pre-fault mode, where the behaviour of one or more units (i.e. thermocouples) show small deviations from normal mode, and c) one or more units are non-functional. To this end, the MCI algorithm has been developed, which is able to detect pro-active incidents during the operation of the reactors.

Related Publication: T. Vafeiadis, S. Krinidis, D. Ioannidis, D. Tzovaras, C. Ziogou and S. Voutetakis, “Real-time incident detection: An approach for two interdependent time series”, European Signal Processing Conference (EUSIPCO’16), Budapest, Hungary, 29 August – 02 September 2016

- IoT Infrastructure
a) UWB localization infrastructure: The objective of developing a UWB localization infrastructure is to provide accurate real-time workers positioning enabling the safety management in industrial environments. Following this objective, a UWB-based wearable device (tag) has been designed and developed able to provide accurate localization. In addition, this wearable device is able to support ergonomics, for instance, by monitoring the trunk posture of the worker. Due to the harsh industrial environments, a robust indoor localization algorithm has been developed that runs on the tag itself. The overall algorithm has been proved robust to manufacturing environment by combining UWB radio communication and inertial sensors. Furthermore, a second, miniaturized prototype has been also developed.
b) Single & multi-radio nodes for collecting data from shop-floor sensors: The objective of developing single and multi-radio nodes is to add robustness in the IoT communication. This is motivated by the fact that the performance of wireless communication is heavily affected by metallic objects and interference present industrial environments. Since some sensor data contain information which could help making decisions in emergency situations, reliable single and multi-radio nodes have been introduced to operate in harsh industrial environments. Thus, the single-radio node is based on the 6LoWPAN communication protocol. The robustness in single-radio node is achieved by exploiting a frequency agility module. Furthermore, multi-radio nodes have been designed
and developed as well for the harsh environments to guarantee a higher degree of reliability not achievable by the single radio nodes. These nodes are designed to provide multiple interfaces, Wi-Fi and 6LoWPAN.
c) Comfort-related IoT devices: A network of sensors measuring environmental parameters affecting the comfort level of workers has been designed and developed. Ambient temperature, humidity, light and sound pressure levels are measured periodically in order to categorize areas as comfortable for workers or not. Lightning and air conditioning control based on measurement is also considered as an option. Measuring nodes are either available in a Wi-Fi based mains powered standalone version or a ZigBee based sensor network with battery operated nodes w

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

"The key innovation points of the project can be summarized as following:
- AR glasses software, firmware and hardware
One of the key innovation points on GLASSUP’s AR Glasses is that they have been designed for an industrial environment from the beginning. Thus, the protection and the ergonomics of the eyewear are a primary factor and this is the only AR eyewear known based on this philosophy.
Also, one of the activities that will be completed in the next period is the support for a thermal camera that will be shown directly on the device, a sensor that has never been integrated in any AR device available on the market.
Another interesting innovation is based on the fact that we are providing a REST API to connect directly to the eyewear and an SDK for IOS, Android and Windows.

- AR "on-job"education & training toolkit
The expressive power of the Standard Operating Procedures Model is really a significant plus and it is – no more, not less – a redefinition of the word: procedure as used in an industrial environment. It has the ability to describe arbitrary procedures allowing to distribute them as a digital, versioned, rich-media, consistent bundle that can be downloaded and transmitted in an single transaction. Finally, The capability to incorporate external sensors readings into a live training / working session makes the tool able to test the trainee in real time with ad-hoc settings.

- Algorithms and technology for employees localization
The design and development of the localization algorithm for employees has been done for the UWB-based wearable device presented is section 1.2. Since the wearable device needs to operate in harsh industrial environments, an innovative robust indoor localization algorithm based on a Bayesian method has been designed and developed. In order to make the localization more robust against the numerous interferences, a hybrid and cooperative localization has been designed. As a first step, the designed algorithm was focused on a distributed hybrid algorithm based on both UWB radio technology and inertial sensors. The cooperative approach along with a TDMA method, for avoiding collision between the UWB wearable devices, will be developed in the second half of the project.
Inertial sensors have been adopted to detect the acceleration, the angular speed and the orientation of the moving target by using its sensors (accelerometers, gyroscopes, magnetometers); note that these are additional information that cannot be observable by using only the UWB radio. These observations in combination with the UWB-based ranging measurements improve the localization performance because they add a full knowledge of the tag kinematics. In addition, the inertial sensors are able to provide the localization estimation even when the ranging is not available or the number of measurements is not enough for estimating the position.
The localization algorithm consists of three parts. Part 1 checks if the inertial measurements have to be processed or not. This choice depends on the level of disturbances and if the acceleration of the worker is zero or in static conditions without disturbances. Part 2, which uses the measurements received from part 1, implements an Attitude Heading Reference System (AHRS) that estimates and corrects the errors of the estimated worker’s attitude and the bias of the gyroscope. Finally, part 3 implements an Extended Kalman Filter (EKF) based on a Position-Velocity-Acceleration (PVA) model that performs the data fusion task of the hybrid localization. In particular, it estimates both the location and the bias of the accelerometer taking as input: the estimated and corrected attitude, the acceleration in body frame and the UWB ranging.
In particular, the EKF AHRS and the EKF PVA were developed and tested as separated modules. However, due to some problems related to the available RAM of the wearable device, it was not possible to test the complete designed algorithm; instead of that, a simpler version of the hybrid algorithm, which includes the EFK PVA and the computation of the Euler angles, was tested. In the second half of the project, the above memory problems will be sorted out and the development will be fully completed and tested.

- Algorithms for gesture and content recognition
Gesture and content recognition algorithms have been implemented for the needs of the Gesture and Content Recognition Manager. Image processing methods have been developed such as object detection (e.g. helmet, jacket, etc.) and recognition using template matching, face detection, skeletal tracking etc. utilizing both RGB and depth information. Furthermore, for each object under interest shape descriptors (e.g. image moments, image central moments, and Hu moments) have been utilized, while clustering algorithms (e.g. Radial Basis Function SVM) have been developed detecting the presence or not of the object under interest (e.g. helmet, etc.).
Furthermore, skeletal tracking algorithms have been developed, where skeletal features (e.g. position, height, speed, motion, etc.) are extracted. In order to detect human falls, all these features are extracted, tracked, analysed and combined.
The proposed algorithms have been tested in real-life environment at CERTH/CPERI’s shop-floor. The expected impact is to evolve the shop floor level to a more secure environment by deploying tools that facilitate daily operations performed by workers while enhancing security mechanisms to propagate and detect information about potential hazards in more punctual manner.

- Advanced Signal processing for early incident diagnosis at factory processes
1) A real time, multi-camera, multi-space system is implemented for incident detection with depth cameras. Once depth images are acquired object detection and tracking is achieved. Features of the tracked objects that characterize the events of interest, i.e. falls, collisions, falling
items, intrusions to restricted areas, are extracted and fed into three state Hidden Markov Models that recognize the events.
2) Thermal information processing algorithms are performed at the implemented incident detection engine in order to segment regions of interest within the incoming frames. To this end, a sequential methodological framework is followed consisting of foreground extraction, contrast enhancement, probabilistic superpixel segmentation, and MRF-based superpixel labelling in terms of sound and potentially defected areas. The proposed methodology is tested against datasets from both CPERI’s and SUNLIGHT's shop-floors.

- IoT Infrastructure to enable context-aware analysis of the shop floor activities
The IoT infrastructure consists of: single and multi-radio sensors (implementing the robust communication), Ultra-Wide Band based wearable devices and comfort sensors based on ZigBee and Wi-Fi protocols.
The GW of the robust single radio node was implemented on a Raspberry Pi 2. The GW handles the operations like spectrum sensing and frequency agility features. The robust single radio node was developed on the Zolertia z1 platform (integrating its on-board temperature sensor) and uses the 6LoWPAN protocol to transfer data.
The Comfort sensors based on ZigBee and WiFi have been developed in order to measure environmental parameters affecting worker comfort levels. Ambient temperature, humidity, light, and sound pressure level sensing components have been integrated on a single ultra-low power battery operated ZigBee node offering months or even years of operation on a single lithium battery cell. A standalone mains power Wi-Fi version of the node has also been developed to offer a lower cost alternative for cases where a sparse sensor deployment is required and existing Wi-Fi infrastructure is available to support the sensor network.
For the multi-radio communication, both GW and multi-radio node were developed using Raspberry Pi 2. The multi-radio node handles functions like network monitoring and interface selection. While the frequency agility features related to 6LoWPAN are managed by the GW.
The robust single radio node has been deployed and tested at the CERTH/CPERI shop floor. The resulting average delay was 31.21ms. The packet loss is calculated in consecutive time intervals, each of them with duration equal to 20 seconds (including 20 transmitted packets). The overall packet loss rate, considering the whole duration, is equal to 0.12% that represent a promising results compared to the performance of commercial wireless sensors.
Concerning single and multi-radio nodes, security features both at network layer and transport layer have been investigated. These security features will be added using Contiki-OS that has been already used for the firmware development of single and multi-radio nodes. These security mechanisms will be added in second half of the project.

- Gamification toolkit for industrial environments
The single gamification elements, which are applied in SatisFactory, are not scientifically new. Gamification artifacts are already elaborated in current research. The innovative aspect of SatisFactory is rather the particular combination of elements, which are carefully chosen for the project pilots sites, based on the user-centered design process.
The biggest innovation factor is, however, the application in a real-world scenario. Gamification has rather been a research topic so far, but there are few examples of successful application in real scenarios. Most of these applications are restricted to single gamification elements and many are used within web tools. However, the proposed application of a holistic approach to a factory setup is novel. If this proves to be successful, there is possibility for a general approach of improving worker productivity and at the same time improving the attractiveness of the workplace for the workers - two things, which are often enough contradictory."

Related information

Record Number: 194894 / Last updated on: 2017-02-16
Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top