Final Report Summary - PROFITEX (Providing Fire Fighters with Technology for Excellent Work Safety)
Executive summary:
The aim of project ProFiTex was to support fire fighters in their perilous work with a system that supplies mission-relevant information without overwhelming the fire fighter. The system comprises two main components:
1. A fire fighting jacket with integrated electronics and a braided security rope, called Smart LifeLine which is able of transporting data. The jacket comprises electronic devices like an infrared camera, movement and localisation sensors and a volcano button integrated into the jacket sleeve. Wire-bound data transfer are be used since wireless communication is difficult over long distances and through several walls of a building. A security rope carried by fire fighters during a mission shall be equipped with data transmission capabilities. This allows information to be sent outside to the command post and back to the fire fighter. By monitoring several parameters of the fire fighters condition like his movement pattern and stance, problems can be detected immediately. The fire fighter himself is supplied with the possibility to navigate even in smoky environments thanks to the infrared camera and the positioning system implemented into his equipment.
2. Information will be displayed to the fire fighters, their group leaders and the commander outside the building. The amount and type of information supplied are being carefully chosen, considering the physical danger and psychic stress fire fighters are opposed to. Work safety of fire fighters shall be increased, thus lowering the number of work-related accidents and casualties. Fire fighting missions will be more efficient using the system.
Project Context and Objectives:
Fire fighting is a perilous job which brings people and equipment to the limits. Each year over 5.000 deaths are caused by fires in Europe alone, and among these dead are more than hundred firefighters. Providing technology for the professionals who fight these fires is the mission of the EU-funded project Profitex (http://www.project-profitex.eu).
Usability and user-centered design are a key point in the project. There are different roles and requirements in a fire-fighting intervention:
- The firefighter himself inside the building;
- The group leader, monitoring several teams of firefighters in the building;
- The mission commander, located outside the building, coordinating the intervention.
Project Results:
Description of Work
Task T 1.1 - Workplace Studies
In T 1.1 the consortium will conduct empirical studies of the working conditions of firefighters as relevant for the project's RTD objectives. The focus of these studies is on the actual characteristics of fire fighting operations but there are also other relevant aspects such as professional training, assessment of new technologies and adaptation of operational procedures within fire fighting organisations. The objective of these studies is to establish an empirically grounded understanding of:
- The operational constraints for the systems to be developed (the what)
- The constraints for an effective participation of the available firefighters in the design process and for the effective appropriation of the developed technologies into operational use (the how)
Results achieved in WP1
PROFITEX adopted the following focused objectives regarding workplace studies:
1) validate existing results and adapt where necessary;
2) create shared understanding among PROFITEX partners on firefighting work practice;
3) conduct complementary workplace studies where this provides substantial benefit.
This exercise was covered by the German TV channel 3SAT for their science program hitec: http://www.3sat.de/page/?source=/hitec/158804/index.html
Description of Work
Task T 1.2 - User Requirements
In T 1.2 will elicit user requirements focusing on the work situations identified and described in D 1.1. These user requirements are distinct from system requirements in that they do not imply any particular solution but simply state the problem or challenge for which the users, here the firefighters, require a solution. This task represents an important analytical effort to understand the actual needs of fire fighting operations in order to guide the search for and the assessment of possible solutions without focusing on any particular solution too early. Of course, the limited resources present in projects such as Profitex require focusing efforts on one or few solutions eventually. For the user requirements, the workplace scenarios described in D 1.1 as well as the project's technological capabilities described in D 2.1 will help in focusing our efforts.
T 1.2 will report the user requirements in D 1.3.
Results achieved in Task
These requirements were written with respect to Volere Requirement standards. This requirement standard distinguishes itself from other requirements standards by testing the requirements while they are being written. This is carried out by looking associating a fit criterion with each requirement. This fit criterion is used to determine if the proposed solution can fit with this requirement or not.
Description of Work
Task T 1.3 - Prototyping and Participatory Design
In T 1.3 the consortium will engage firefighters and other stakeholders in a participatory design process. This activity is based on the understanding that usable and eventually accepted designs do not result from the implementation of specifications that have been obtained before the design phase. Instead, successful designs result from many iterations of putting the artefacts to work in the hands of the users and obtaining their feedback. To enable this process even before functional system prototypes become available, T 1.3 will prepare and organize a series of workshops where the use of systems or system components during fire fighting operations will be simulated with different means, based on the prototyping approaches developed in the wearIT@work project. In particular this includes simulating fire fighting operations with the FireSim prototyping tools, allowing multiple actors to use virtually simulated future technologies in either a fully virtual or a mixed-reality environment, i.e. an augmented physical training facility or a virtual building. The purpose of these activities is providing to the firefighters as rich as possible an experience of using future technologies in realistic usage situations and thus enabling them to provide design-relevant feedback in a meaningful way.
Results achieved in Task
In addition to a continued improvement of the FireSim prototyping platform, from version 1.3 at the midterm review to version 1.6 at the final review, with virtually hundreds of bug fixes and improvements, two major additions were made during this time. First, the Command Post System prototype was implemented as part of FireSim, enabling interactive testing with very limited effort and very shortly after the concept had been defined. This implementation was also used in the studies reported as part of T1.1.
Description of Work
Task T 1.4 - System Requirements
In T 1.4 the specific requirements for a system addressing a defined set of user requirements from D 1.3 will be described. The first step here is to select a coherent subset of requirements from D 1.3 that Profitex will address. This selection will depend on the project's feasibility evaluation regarding different technological options in WP 3 and a prioritization of requirements by the firefighters. Similar to the user requirements, the system requirements will not specify a concrete implementation but provide measurable criteria for judging whether any given system meets the requirements.
Results achieved in Task
Task T 1.5 - Economic analysis and financial planning
With the aim to develop an economic analysis which supports assumptions made on demands in the short- and medium-term, TexClubTec will provide market data and data of potential customers and fire brigades.
Results achieved in Task
Within this task potential customers of Profitex were identified. Beside fire brigades TexClubTec (TCT) identified army, police, national and regional civil defence organizations and industrial companies with own internal fire fighter services. The army and police need a different configuration of e.g. sensors to be used in their application scenarios. Search & rescue scenarios in unknown and dangerous environments can be transferred immediately. Also the concept of the information centre - the command post - is very useful for the other target groups. While the main focus of a special force of the police or army is not fighting fire but rescuing people out of dangerous situations and neutralizing one or more suspects in not predictable areas, the Profitex system enhances army and police operations due to the advanced information technology implementation.
Description of Work
Task T 1.6 - User Reference Group: Workshops and Online Community
T1.6 deals specifically with organizing an exchange of knowledge and experience among the members of the User Reference Group on topics relevant to the Profitex project. While the aim of T1.3 is to engage end-users in hands-on design activities one organization at a time, T1.6 focuses on bringing the different perspectives from end-users together by fostering an open debate among the members of the User Reference Group. This exchange will be facilitated by two principal means: 1) dedicated annual workshops where representatives from all end-user partners as well as the technical partners will work on principal aspects of the project and 2) an online community where a continuous exchange is motivated and facilitated, focusing more on a sustainable lively discussion than on the exchange of abstract information.
Results achieved in Task
The realization within PROFITEX of not being able to develop functional physical LifeNet beacons let to a re-evaluation of the Reference User Group (RUG) strategy. As explained above, the inability to develop LifeNet beacons within PROFITEX required the search for an alternative localization technology and the adaptation of the navigation support system to this technology. To this end, a more targeted technical discussion was required which was only practical within a smaller group of end-users and domain experts familiar with challenges faced and available to engage in a sustained technical discussion. Therefore, this discussion was conducted with fire fighters from HEAT and other domain experts from within and external to the PROFITEX project.
Work Package 2 - Technology Development
Objectives
Within this work package the planning and design of the single components will be done. Starting from a definition the components will be planned. Furthermore the components will be manufactured, and in combination with WP 4 they will be tested. In the end a decision about which components will be used for the next steps will be made. This will finally lead to a system design, that can be tested under real conditions.
Description of Work
Task T 2.1 Definition and planning of single components
The following technical parts are essential for the whole concept.
Power Supply System
- Definition of a modular component system
- Evaluation of the data volume
- Definition of internal (jacket based ) interfaces (Textile Bus System)
- Definition of the jacket to lifeline interface
- Definition of buttons and display (HID)
- Definition of system behaviour in case of system failure
- Security aspects ('is there any danger coming from the system?')
- Definition of performance
- Results achieved in Task
Requirement plan
Protective clothing is closely related to the clothing sector. Potential solutions and product designs are complex due to their intended purposes. Considering protective clothing for firefighters there is mainly the task of heat protection and the transmission of sweat. To fulfill all these tasks standard fire-protective clothing consists of several layers. The layer system depends on the specific model of a garment.
Basically it is structured into
- Outer shell
- GORE-TEX membrane
- Inner liner
Requirement plan
In order not to endanger firefighters, protective clothing must meet the following requirements.
- Thermal Protection
- Moisture and hot steam barriers
- Re-drying and sweat permeability
- Weight and ergonomics
- Visibility
- Robustness
- Haptic
Description of Work
In this task all components are manufactured by the technology partners. If necessary, not only one technical layout will be manufactured, in certain cases also two or even three different approaches might be necessary.
Results achieved in Task
Smart Lifeline Development
The development of the Smart Lifeline was done in the course of a PhD thesis. All images and results describing the Smart Lifeline development are taken from [Eic12] with the author's permission.
Conventional data cables
In a first step regular data cables have been analysed and their main components were identified. An electromagnetic shielding (in this case a metal-coated polymer foil) is wrapped around the cable. The outer layer is formed of an extruded polymer jacket.
Smart Lifeline Concepts
The Smart Lifeline has been developed in two variants. Variant A consists of a textile data cable with no electronic devices in between the two ends of the cable. The main goal in the design and production of this data cable is to research the integration of data lines in braided structures. Critical parameters like cable resistance and signal transmission quality are to be assessed with this prototype.
The components of the Variant A prototype are as follows:
- Core rope (for mechanical stability)
- Functional layer (holds the electric leads
- Insulation layer (separates the functional layer and the EMI shielding layer)
- EMI shielding layer (against electromagnetic disturbances)
- Mantle layer (mechanical protection).
The elements in this design are as follows:
- Core rope (for mechanical strength)
- Woven narrow fabric with electrical leads
- Adaptor, bottom shell
- Adaptor, top shell
- Electrode channels (here the electrodes of the electronic device can be plugged in)
- Insulating braiding layer
- EMI shielding layer
- Mantle layer
- Electronic device
Measurements procedures
After producing the Smart Lifeline prototypes each one was tested considering its signal transmission capabilities. The Smart Lifeline Variant A and B were subjected to the the following tests:
- Laboratory tests (Variant A and B prototypes)
- Insulation test (testing against short-circuits)
- Ohmic resistance measurement (DC resistance)
- Impedance measurement (AC resistance)
- Signal attenuation (signal loss from one end to the other)
- Cross-Talk
- Practical tests
- USB Data transmission test (variant A only)
- Video signal transmission test (variant A and B)
- displays the equipment which was employed during the tests.
Insulation test
Two variants of the Variant A prototype were produced. The electrical leads in the functional layer were realized using four Shieldex yarn (235 dtex) in one variant and copper wire (d = 0.15 mm) in the other variant. An insulation test was performed. This was done by testing the connectivity between 1A-1B-1C-1D-1M. Only 1A is supposed to show a connection. If any other pairing shows an electrical connect, a short-circuit has occurred. Next 2A, 2B, 2C, 2D, 2M are tested and so on.
Ohmic resistance measurement
Cross-talk
Cross-talk is a phenomenon which occurs between neighbouring electrical leads. A signal flowing through a conductor induces a signal in the neighbouring lead, resulting in a signal loss in the signal line and noise in the passive line.
Practical test 1: USB data transmission test
To show the capability of the braided data cable (variant A) USB connectors have been soldered to the cable. USB contains four data leads and a mass lead, same as the variant A prototypes. Therefore variant A prototypes have been confectioned with USB type A and USB type B plugs to connect a notebook and an external harddrive.
Practical test 2: Video signal transmission test
With both the variant A and B prototypes of the Smart Lifeline a practical video transmission test was conducted. The signal was sent from a source (DVD player) to a TV by using an analog composite video signal feed. Composite video requires only two eletrical leads - one for the signal and one for mass. The variant A prototype was confectioned with cinch connectors. The four data leads were all soldered to the signal pin of the cinch connector, the EMI shielding layer was connected to the mass.
Insulation test
The insulation test was performed on both the variant A prototypes (braided data cable with wither copper wire or silver-coated Polyamide leads). The prototype with copper wires exhibited no short-circuits, all four leads and the EMI shielding are clearly separated.
Ohmic resistance, Impedance, Signal attenuation, Cross-talk
For comparison all measurements have also been performed on a standard Cat-5 Ethernet cable. Since signal attenuation and crosstalk are effects which depend on the signal frequency, these measurements were taken at 100 kHz, 1 MHz, 10 MHz and 50 MHz each.
Practical test 1: USB data transmission test
Two variant A prototypes were outfitted with USB type A and B plugs, one variant A prototype was confectioned with USB type A-A plugs.
Practical test 2: Video signal transmission test
The images from the source (DVD player) could be transmitted clearly, there were no disturbances in the video image. This demonstrates that the textile data cables are capable of transmitting a clear video feed.
Beacon Demonstrator
Extensive research on local indoor positioning and localization has been done at Labor to verify the possibility to integrate a powerful computational unit into the SLL beacons and able to provide global and/or local position information with high resolution and accuracy.
Hardware
The development platform previously selected for the Beacons development was dropped once the localization requirement was taken out from beacons specifications. A more simplified architecture and hardware was then chosen.
Firmware
The firmware flashed into the Beacon is able to calculate the retreat path along a series of beacons using an algorithm based on Dijkstra calculation. It also sends and receives messages to the Host coordinator, the TMS1 in the Display, and records the positions of all the other beacons deployed in the GUI software. It also drives the two Green and Red LEDs to signal the Fire Fighter proximity (Red Led) and the route by flashing the Green Led. There is also a local temperature sensor constantly monitored by the CPU; once the temperature is above a preset threshold, the beacon is automatically excluded by the retreat path and each Beacon modifies its routing map accordingly. The routing map is also updated every time a fire and/or a broken path is active in the scenario. All the communications are managed by the ZigBee Module and coordinated by the CPU.
GUI Software PC
The GUI interface is designed to simulate the deployment of a number of beacons in a SaR mission and test the real beacons behaviour when different dangerous situations take place like the presence of a fire, a connection path between two beacons is broken or when a beacon senses a high temperature over a certain threshold.
Fire Fighting Jacket Development
The main goal is to investigate a textile based interface for the fire fighters' protective clothing. It can be used for communication between fire fighter and corresponding system enabling to transmit data to both the fire fighter inside the building and the group leader outside the building. The fire fighters' garment is the medium for integrating the interface and electronics to build up an infrastructure to distribute data and energy. In that way, it will be possible to locate the fire fighter. The data will also be used to monitor the posture and activity of the fire fighter (stance recognition, activity index).
Plan for the Textile Keypad
When combining the HID with a standard fire fighters' garment it does not mean to destroy the garment's functionality, but to full fill the exact statutory provisions and norms. For instance to reach the thermal protection of the fire fighters exposed to 1000°C and to maintain a constant body temperature heat regulation mechanisms with re-dryable and sweat permeable materials.
Structure of the HID
The HID is realized in three devices which are placed at the forearm at the fire fighter garment. Every device is composed of the following:
- water resistant cover protecting the conductive materials against influences like extinguishing water from outside
- relatively broad and wide knitted spacer fabric to get electronic contact by pressing the device (even wearing thick fire fighters` gloves)
- button matrix with the opportunity of nine contact points
The following technology is used for building up the textile matrix switch. The HID consists of three layers.
Water resistant cover:
To protect the electronic components from outside influences there is an upper functional layer. This water-resistant cover is made out of waxed awnings to avoid influence on the from outside such as water and heat. To produce it there are two waxed awnings sewed together. One of them forms a quadratic button whereas the other part forms the ground.
Spacer Fabric:
This functional layer is followed by a spacer fabric to ensure a high material flexibility of three-dimensional feature: spacer fabric which is manufactured by Müller Textil which has lot of experience in the textile warp knitting technology. This fabric is 10 mm thick.
Button matrix:
The third layer consists of a button matrix including silvercoated polyamide (PA) yarns on the inside of the buttons: button matrix. The button matrix consists of two fabrics. By rotating the bottom layer around 90 degrees, a matrix can be formed. The bottom layer has stripes of conductive yarns that connect through a tape with wires (special narrow fabric from OFFRAY®) with the USB-connector. A housing of the electronic interface is needed to encapsulate the electronic and the cabling.
The button matrix is structured like following:
- Upper Function layer
- Knitted fabrics
- Knitted fabric with several stripes of conductive yarns
- Bottom Function layer
Process
step Tools which are needed Main Process Textiles which are needed pictogram
1 Scissors Cutting 2 Knitted Fabrics
- Non-woven
- Fabric with several stripes of conductive yarns
2 Hot iron (140°C) Bonding Fabric
- Non-woven
- Fabric
- Knitted fabric with several stripes of conductive yarns
- Unformed Upper function layer
3 Former plate, Punching tools Thermoforming Unformed Upper function layer
Upper function layer
- Bottom function layer
The bottom function layer consists of the following three fabrics containing diverse properties:
- Woven fabric with several stripes of conductive yarns
Several paths are tightly woven with silver coated conductive yarns into a textile, produced with PES-yarn on the web loom.
- Supporter fabric
The upper function layer can be attached to the supporter fabric to guarantee the required stability for the keypad.
- Awning fabric
Apart from being a supporter fabric this awning fabric also offers additional textile stiffness for the keypad and a hard ground.
Position of the human-interface device inside the fire fighter jacket
Generally the layer system depends on the model of the jacket. The layer system is structured like following:
1.) Shell
2.) GORE-TEX membrane
GORE-TEX membrane is used to guarantee the waterproof despite it is breathable so that it prevents the fire fighters against wind and super heated steam.
3.) Interlining
The HID in form of an additional layer should be placed between the GORE-TEX membrane and the interlining. This way the HID is protected against influences like extinguishing water and rain as well as the high temperatures from outside.
Materials and methods
One of three layers is a knitted fabric with several stripes of silver coated conductive PES-yarn inside. To get the form of the button matrix the knitted fabric including the conductive stripes has been heat set with a heated press machine like mentioned above.
The test apparatus has the following characteristics:
The tactual finger is a steel pin with a diameter of 10 mm coated with foam which is driven by an electrical motor and operated with a sinusoidal output.
Parameters Technical details
Cycle sinusoidal form
Tactual finger Steel pin with coating
Underlay Foam PUR F46
Switching voltage 5 Voltage
Measuring frequency 200 Hz
Pulse frequency 0,7 Hz
material Button matrix
Results
Every button has been tested on its pressing stability for more than 80.000 switching cycles and have been finished manually. It should be noted that the thermally formed button being tested has to retain its switching function over the testing time. But the surface and material stability of the buttons become severely affected after this high switching numbers which has an effect on sagging of the thermally formed three layered knitted fabrics.
The aim of both tests was to provide a relation between human sensing and the influencing parameter
- Thickness of surface material (number of layers)
- Height of transparency curvature (mm): The height of transparency curvature makes the difference between a soft and hard switch.
The haptic tests have been carried out as a questionnaire. The results of the haptic tests are related to the pressing force tests.
- 2- and 3-layered surface with high transparency curvature (> 4,3 mm) possess a pressure point Surface created with more than 3 layers are too stiff
- Low transparency curvatures (< 3,6 mm) do not achieve enough counter pressure
- Sensing of a tactile feedback with the Transparency Textile Compound Switch in contrast to the Textile Switch.
Conclusion
Although the HID worked well under defined Lab Conditions, the function of the buttons gets worse and also the tactile feedback of the user was poor in real user tests. Like we set up before the major criteria is to have a tactile robust and serious feedback and a rugged electrical contact. It is essential that the fire fighter is able to feel a feedback coming from the button. This textile buttons worked well under defined Lab Conditions, but while using them in simulations they turned out to be not yet reliable. Due to the fire gloves this was a serious problem. Also the ruggedness was too weak. The major criteria were to have a tactile a functional button in the meaning of having a rugged electrical contact and also a robust and serious user feedback.
Manufacturing of single components (volcano button)
Due to the experience of the tests we decided to develop a new version of buttons. It is essential that the firefighter is able to feel a feedback coming from the buttons. We also tried to create a feedback loop using symbols in the display, but this was not usable under fire fighting conditions. Also a acoustical feedback, (as known from mobile phones) is not an option due to the noise in fire fighting situations. Most important issue was to find a design that avoids unwanted release. So the main issue of this design was to combine the tactile feedback and a kind of protection boarder to ensure that the user will not activate the button unwanted. This is very important because this would affect the systems performance dramatically. The lead to a design we call 'Volcano Button'. The name is coming from the shape of a volcano, which is pretty the same shape as we realized for the buttons. The Button is a assembly of three parts.
The following drawing shows the construction of this assembly.
It was very important to find a design that fits to the needs of fire fighting applications. This means that we have to find the correct dimensions for the volcano button design. We evaluated a lot of different fire fighting gloves and this leads us to a button dimension of approximately 30mm. The first thing was to find a water tight button (IP67) witch have to be electrically solid and also heat resistance.
This finally results in the following parts:
The next important thing to connect these buttons to the FFS unit was to find an Interface. Therefore it was necessary to convert the bush button signals to USB commands. We used a special electronic for doing this. It is a electronic called AVR Stick.
Description of Work
Task T 2.3 Integration into lab examples
The single components are integrated into textiles. The choice will be made based on the results of the testing in T4.2 The textile material is equal to the material of the end product. Electronic components are connected with textile data energy buses to demonstrate and verify the data and energy transmission.
Results achieved in Task
The main focus of the 'Profitex Firefighting Jacket' will be the practical functionality of the jacket. A general textile integration of the electronic device is possible.
Material composition
The 'Fire Breaker Action Jacket' contains a liner system with the following parts: antistatic outer Shell, Gore-Tex Membrane and Lining. 'Liner system' means that the membrane is placed loose inverted between outershell and inner-lining. It's not laminated with the outershell. This is a big advantage for having the best and easiest possibilities in integrating all needed components.
Gore-Tex Airlock Membrane:
Water tightness and high thermal protection are given due to the intact and breathable Gore-Tex Airlock membrane. The membrane also offers advantages like high protection against contaminated liquids and also high reduction of weight.
Manufacturing
Integration of the necessary technological components into the manufacturing process is possible. Always one component gets modified or changed, we have to adapt the steps in manufacturing. Main important for all developments is of course the functionality in practice. An important issue to discuss is the washability of the jacket. Integrated electronic parts, which are not resistant against water, have to be removed beforehand and integrated again by trained staff. This means additional work and expense, also additional service costs. Efficient solutions have to be found. Also the wearabilty of the Profitex Jacket has to be evaluated in practical experience.
Task T 2.4 Evaluation and drafting of a textile integration standard
The components are evaluated considering their mechanical properties. The compound textile-electronic component is characterised. Compiling the experiences of the textile integration, a standard for the integration of electronic equipment into fire-protection textiles will be drafted. Considering the future production of the Profitex system, production methods and production costs will be evaluated in this step as well.
Deliverables will be:
D2.8 Report on the evaluation of the system design
D2.9 Standard for the integration of textiles into fire-protection components
Results achieved in Task
A certification for the Profitex jacket has not been done so far, as this certification would also include the certification for the electronic integrated devices and components.
Description of Work
Task T 2.5 Signal-processing for components
VUT and ETH have experience and ongoing research projects in pattern recognition and signal processing to interprete and analyze data from bio-sensors.
Results achieved in Task
Bio-sensing
For an 'everyday life and field application', a minimal sensor setup is desired for comfort reasons.
The monitoring belt consists of three smart fabric sensors to acquire cardiac activity, breathing rate and skin temperature (Zephyr). The body acceleration is measured with a 3D accelerometer included in the recording device. In addition to ECG data, the chest belt provides RR intervals by measuring the duration between two consecutive R waves of the ECG.
Work Package 3 - System Development
Objectives
Creation of a complete textile system
Description of Work
Task T 3.1 Firefighter System
In Task 3.1 the specification, design, development and integration of all components belonging to the system used by the operational firefighters is carried out. This task includes the creation of early physical prototypes to investigate ergonomic options. The firefighter system also includes all augmented tools that might be carried by the firefighters such as the Smart LifeLine.
Results achieved in Task
Fire Fighter System
Computing Unit
Following the results presented in the Midterm report and the decision taken at the Workshop held in Rome in October 2011, a new computing unit for FFS2 has been developed to better accommodate Firefighters needs. The new computing unit has been dramatically reduced in its dimensions, weight and power consumption. The operating system has been switched from Windows to Android. All heat and thermal issues have been washed away with the new implementation.
Hardware
After some research Labor targeted the main board to be the versatile BeagleBoard-xM (BB-xM), a Laptop- like performance system based on a low power TI ARM CORTEX A8 with high graphical and computational capabilities.
The BB-xM has been physically modified and integrated with extra custom circuitry, electronics and hardware for battery management, protection and integration with the Computing Unit Case.
The case was designed to be IP68 compliant with rubber rings and perimetral screws on the top cover to guarantee the appropriate sealing.
3D Reconstruction & Tracking and Thermal Mapping
Tracking the location of firefighters in fire brigade operations is one of the main objectives of the Profitex project. In addition to their position in space, it is important for an operational commander to know about the structural context (i.e. building structure) in order to safely guide his men and women during an operation. 3d models or accurate maps of buildings are rarely available before an operation. Therefore, our system has been designed to reconstruct the 3D structure of a building in real-time. The constructed mesh can then be used to provide an operational commander with a better overview of the situation. Simultaneously with the reconstruction of the 3D structure, a fire fighters position and viewing direction is calculated relative to it.
Thermal Imaging
Thermal imaging is important for a firefighter's security in his or her daily work. Firefighters use thermal cameras to see in dark and smoky environments, to search for victims and to find hot-spots and other potential dangers. From literature the importance of thermal imaging for firefighting becomes obvious, for example in (Amon et al. 2005), where a detailed report of a workshop on thermal imaging research needs for first responders is given. Therefore, we have integrated thermal imaging in our live system to enhance informative value and expressiveness of the 3D reconstruction.
SLAM
State of the art approaches give a solution to the problem of self-localisation and reconstruction in unknown environments by realizing Simultaneous Localisation and Mapping (SLAM) with visual data and additional sensors. SLAM has initially been developed in the robotics community and has undergone great advances in the last decade (Bailey & Durrant-Whyte 2006)(Reitmayr et al. 2010). The aim is to simultaneously build a map in an unknown environment and to calculate the objects position within this map, which is very useful if there is no map available. The main approaches to solve SLAM are EKF-SLAM (R. Smith et al. 1990), Fast-SLAM (Montemerlo et al. 2002), and Vision-Based-SLAM (Sim et al. 2005) among others. These approaches rely on information obtained from sensors such as sonar, stereo cameras, laser scanner, etc., and/or odometry. Most of them have been applied for robots, unmanned vehicles or aircrafts. Furthermore, MonoSLAM and the more accurate Parallel Tracking and Mapping (PTAM) system (Klein & Murray 2007) allow to investigate flexible environments with 3D infrastructure.
Depth Cameras and Kinect Fusion
Depth cameras provide input to another class of mapping and tracking algorithms e.g. (R. A. Newcombe et al. 2011). Here, the depth information is not calculated from 2D images, but directly streamed from a sensor.
Thermal Cameras
Cameras that generate thermal images in the long wave infrared range (LWIR) are referred to as thermal cameras or FLIR (forward-looking infrared) cameras. Thermal cameras, operate in the long wave infrared band which refers to radiation with wavelengths in the range of 8-15µm. Each object (surface) with a temperature higher than absolute zero (0K) emits energy with a certain wavelength. The hotter the object the more energy is emitted. The LWIR band captures most of the wavelengths of thermal radiation that we are confronted with in our natural environment. Thermal cameras are able to measure the emitted infrared radiation in the LWIR band and visualize these thermal patterns as an image (Balaras & Argiriou 2002).
Hardware & Setup
In our system we are using an Asus Xtion Pro as depth sensor, which provides depth data from the structured light approach described in subsection 0. The depth data is streamed at a resolution of 640x480 pixel and a frame rate of 30 frames per second.
Operating System
The Operating System installed on the Beagle Board and chosen by the consortium was Android. The appropriate Android OS kernel version has been tested and crafted on the BB-xM to interface the Firefighter System devices:
- Helmet Display
- HMI
- Breadcrumb Wireless communication device
- Bluetooth Health system monitor and PDR
Two versions were tested before choosing the final one: Gingerbread 2.3.4 and IceCreamSandwich 4.0. The final OS installed on the CU for the FFS2 is the Ice CreamSandwich 4.0 which proved to be more stable and faster than the GingerBread version.
Breadcrumb
The Breadcrumb device integration was conditioned to have a separate source battery supply and mounted on the Fire Fighter Oxygen tank. The first proposal sketch is reported below.
The subsequent change to this sketch was the cable arrangement: instead of having a secondary plug on the back of the jacket, it was decided to reuse the SLL connector from FFS1 and run the breadcrumb cable, together with the Helmet-Display cable, along the shoulder strap down to the connector.
Description of Work
Task T 3.2 Group Leader System
In Task 3.2 the specification, design, development and integration of all components belonging to the system used by the group leader is carried out. This system supports the group leader in receiving information from the team for monitoring and planning purposes, in detecting critical conditions, and in taking decisions to secure the safe and efficient evolution of the operation. For example, this system is supposed to provide information on the search status of a given building floor based on the localized information provided through the Smart LifeLine. It is important to note that the development of a completely new hardware device for the Group Leader System is not within the scope of the Profitex project. Instead, the consortium plans to use and integrate available hardware such as a ruggedized TabletPC or UltramobilePC.
Results achieved in Task
Design and Implementation of the 3D-Reconstruction and Thermal Mapping
Kinect Fusion Algorithm
The Kinect Fusion algorithm (R. A. Newcombe et al. 2011) processes every frame obtained from the depth camera and generates two things: an updated 3D structure containing information on surfaces in the scene and the camera pose.
In order to do this, multiple steps have to be executed:
- Preprocessing: From the current 2.5D image, positions and their normals are calculated at multiple resolutions.
- Pose Estimation: Using the Iterative Closest Point (ICP) method the pose of the current frame is aligned with the previous surface data stored in single combined 3D structure - the volumetric truncated signed distance function (TSDF) (Curless & Levoy 1996). In fact, the frame is not aligned with the whole data structure, but only that part of it, where the prediction of step 4 supposes that the frame should be located. Therefore, the fast projective data association algorithm (Blais & Levine 1995) can be used.
- Reconstruction Update: Once aligned, information contained in the current depth frame is merged in the TSDF volume. Over multiple frames the depth/surface values in the TSDF structure are averaged. Therefore, the final TSDF structure contains less noise and holes and is much more accurate than a single depth image.
- Surface Prediction: Based on the last poses of the camera the next pose is estimated and using ray-casting a surface model from that perspective is created from the TSDF volume. This local surface model is then used in step 2 to align the next frame.
From the TSDF volume a point cloud, mesh or other representation of the surfaces in the scene can be finally extracted.
Implementation Kinect Fusion
Our implementation of the Kinect Fusion algorithm has been developed in C++ and is based on the Point Cloud Library (PCL) in large parts. We make use of the available methods implemented in CUDA and optimized for processing on the GPU.
Description of Work
Task T 3.3 Command Post System
In Task 3.3 the consortium will address, with a relatively small effort, the question of how the information obtained through the Profitex systems may be made available and exploitable at an information system at the command post. Similar to the group leader but one or two levels (in case there are sector chiefs) up the hierarchy, the command post is in charge of keeping track of the engaged teams and their missions. Generally today, command posts are not concerned with information on individual firefighters. Nonetheless, support for handling aggregated information is likely to be of interest. And as before, special circumstances such as lost contact to an unresponsive firefighter may warrant focusing the command post's attention on an individual.
Results achieved in Task
The Command Post System in Profitex was realized using COTS devices.
The approach is to provide operational pictures with schematic representations of tactically relevant information (connectivity, tagged locations, responders, hazards). Schematic representations are used to deal with the inaccuracies and sometimes inconsistencies of the available information and the fact that this information becomes available gradually over the course of an intervention.
Description of Work
Task T 3.4 Monitoring the firefighter's activity and health
This subtask is concerned with the implementation of procedures to continuously monitor the firefighter's activity and health during a mission. The resulting indicators of activity and health will be provided to the commander of the mission by using the communication infrastructure that is developed in WP2. In this way, the commander is always aware of the physical and physiological status of his firefighters and can react accordingly.
In order to achieve a continuous monitoring of the firefighter's activity and health, the following working steps will be carried out:
- Selection and adaptation of available sensor technology in cooperation with other technological partners
- Characterization of sensors in terms of signal quality, stability and liability to motion artefacts
- Design and implementation of laboratory experiments in order to build up a sensor data repository
- Sensor data pre-processing and feature computation
- Modelling of activity- and health-indicators based on selected signal features
Results achieved in Task
In the following we introduce the sensing equipment to monitor physiological health status, physical activity and cognitive performance.
Physiological Monitoring
In our experiments we used Zephyr first responder system BioHarness (Zephyr) as the physiological monitoring system as explained in Task 2.5. The Zephyr system is a mobile multi sensor chest belt. It combines physiological parameters with activity measurements and posture analysis. The sensor electronics module (SEM) measures heart rate (derived from electrocardiogram (ECG)), breathing rate, skin temperature, activity and posture data.
Activity Monitoring
For the activity monitoring we have used orientation sensors to capture the body motion and orientation of each fire fighter. The sensors were attached to the head, the torso, the lower arms and the right foot. This not only allows us to measure the activity level of each fire fighter in a fine grained manner for each body part, but also enables us to investigate the coordination behavior of two firefighters in the same troop.
Monitoring of Cognitive Performance
Cognitive assessment is the examination conducted to determine subject's level of cognitive function and skills. In this task cognitive assessment is investigated to identify mild cognitive decline in order to determine the ability to manage complex activities such as search and rescue. Several standardized desktop-based cognitive assessments have been designed in which users have to watch a display and respond actively in the test by using keyboard, mouse or special buttons. These tests measure functions such as reaction time, vigilance, attention, learning, memory and sensory function. Scientists use such tests to assess mental performance; however, they are not feasible to be used in the field. One limitation is that the user has to stop all ongoing activities to participate in testing and another is that the equipment is often not portable. Therefore we designed a wearable device which is able to measure reaction times of a person during activities of daily life in order to assess cognitive functioning.
Activity Level and Troop Coordination
When the troop receives their mission command they start with the search and rescue mission and their activity level increases rapidly (green and red colour). The diagonal trend from the lower left to the upper right depicts the coordination behaviour of the two fire fighters in the sense that the first firefighter is followed by the second fire fighter. It can be seen, that the coordination behaviour changes at the end of the mission when the person is being rescued because now the second firefighter leads the way out and the first firefighter follows.
Physical Workload
We divided the data into three segments (i) baseline (shown in green), (ii) search and rescue (shown in red), and (iii) recovery (shown in green). According to Zephyr data, the mean heart rate of the firefighter was 96 during the baseline period. High value of the baseline HR can be explained by the heavy firefighter gear. During search and rescue mission, the mean heart rate of the firefighter reached to 140 BPM and the mean HR decreased to 119 during the recovery period. The effect of the high physical workload can still be seen on the recovery period since it needs longer time to calm down.
Posture Mirroring
The automatic detection of nonverbal cues such as body movement from sensor data provides a way to capture aspects of human behavior. An important nonverbal named posture mirroring occurs when persons display similar postures while interacting with one another. Research in psychology has linked posture mirroring to increase rapport and empathy and to support communication. This supportive functioning could be important to monitor during fire fighting missions as it would allow commanders to gain an insight into the nonverbal communication patterns between fire fighters throughout a mission. This knowledge could then be used by the commander in post incident briefings and further fire fighter training.
Monitoring firefighting teams during training scenarios
We investigated the use of smart phones to monitor firefighting teams in order to improve post incident feedback and to provide better training. We have explored what basic team behaviors such as team activity and team communication can be measured with the smart phone in the context of firefighting. We rely on built-in sensors of the smartphone because firefighters are used to carry their mobile phone with them even during incidents.
Experiment
The experiment was conducted in the fire simulation building at the training facilities of the Zurich fire brigade. In the fire simulation building a variety of fire scenarios can be realistically simulated ranging from kitchen fires to a burning car in the garage. During training, firefighters are confronted with real fires, extreme heat, high humidity, severely restricted visibility and thick smoke.
We have recorded motion and audio data of 10 civil firefighters during one day of training. In the chosen scenario a kitchen fire in the third floor of the training building had to be extinguished. Two teams consisting of five firefighters each completed the scenario. After the first team finished the scenario, the instructor gave feedback to all firefighters and highlighted mistakes that should be improved, then the second team started with the same scenario.
Team communication estimation
In order to capture the amount of verbal communication during a firefighting incident, we employ the robust voice activity detection algorithm presented in [1]. At its core a long-term signal variability (LTSV) measure is used to measure the degree of non-stationarity in the audio signal and is hypothesized that speech has a higher degree of non-stationarity as compared to noise sources. Because the LTSV-measure is independent to amplitude scaling of the input signal it is robust and well suited for the harsh firefighting environment. We tested the detection accuracy by manually annotating speech of four firefighters during the training scenario and obtained detection accuracies between 75% and 85% even in high noise conditions.
Body and speech activity
Body and speech activity describes the fraction of a two second window that a firefighter was active. The body and speech activity level of the two teams is displayed. When comparing the two teams it becomes obvious that team 2 was faster than team 1. When comparing the body activity of the engineer it becomes clear that the engineer in team 2 was much more active than the engineer in team 1. Concentrating on the preparation phase we can infer that team 2 appears to be better coordinated as all members are first active and before they continue to work, first stop to communicate which can be seen by more speech activity between the troop leader and the incident commander. This pattern is notably absent in team 1.
Conclusion
We have demonstrated that the measurement of important team behaviors such as the amount of communication and the mount of body activity can be measured with the smartphone in a typical firefighting scenario. Moreover, we showed the potential how the measures could explain why one team was faster than another. In future research this needs to be validated with a larger sample size to proof generalizability.
Description of Work
Task T 3.5 Visualization of video data
Enhancing the video signal from the worn infrared camera is vital for the firefighter to make the correct decisions when confronted with dangers within his environment. The image data from the video signal is a grayscale presentation of the temperature distribution in the surrounding scenario. To assist the firefighters in a very stressful situation the grayscale values of the image can be mapped to different color schemes to give the firefighter a more meaningful picture of what is happening around him. These mapping techniques are common practise in volume visualization. In medical application different tissues of the human body from scanned CT or MR data are emphasized by so called transfer-functions. Thus here it is possible to enhance the grayscale video image by highlighting humans and emphasizing objects in the surroundings that exhibit a potential risk for the firefighters, like objects that exceed a specific temperature which are dangerous for them.
Results achieved in Task
First tests with our system in different situations showed good results. We were able to reconstruct different large scenes like a building entrance, different rooms, hallways and staircases with a level of detail, which should be more than sufficient for the proposed purpose.
Thermal imaging with calibrated temperatures enabled us to clearly highlight human body temperature in environments with room temperature. Our experiments with thermal imaging depicting relative temperatures suggest their usefulness even in situations, where the temperature differences are smaller.
Limitations and Future Work
We have not yet tested our system in dense smoke, but from the technologies used, it has to be expected that performance will be significantly hampered by anything blocking or degrading the quality of a direct view in the visual and especially in the near-infrared spectrum. In environments densely covered with smoke it is very likely that reconstruction is brought to a halt. In this case a user could, however, still revert to thermal imaging to retrieve information about structures and people in the building.
Description of Work
Task T 3.6 Integration
In Task 3.6 an integration plan is defined and executed to ensure that all partial systems developed in WP 3 can successfully form one interoperable system. Experience shows that this integration effort requires dedicated responsibility within RTD projects and that it needs to be addressed not after the components become available but ideally before development starts.
Task 3.6 will result in an Integration Plan (D 3.13).
Results achieved in Task
Integration Plan
All cables which are necessary for electronic devices are hidden and not visible for the wearer of the jacket. Only some cables, which belong to the CPU and go down to the bottom of the jacket, are visible.
Cables are fixed on the inside with a velcro loop to avoid moving.
Materials
In the beginning we had doubts regarding the implantation of the textile integration. Cables should go from the outside to the inside of the jacket, without damaging the membrane. Integrated electronic device should not be in contact with heat or water.
Production
Another difficulty is the integration of all electronic devices during manufacturing process; it's not possible to integrate it to the finished jacket. Our dressmaker in the manufacturing plants would have to work with new materials and accessories. They would have to get the feeling for the electronic parts and work very carefully, as the fabric could be damaged while inserting the electronic device. They would have to be trained regarding cable routing and special handling of each piece of electronic. We would need additional staff for production of Profitex Jackets, which would be in charge of this only, to avoid mistakes.
Delivery
We have to think about special deliveries. A jacket with electronic device needs a special package, a cardboard with padding material inside. The receiver of the delivery has to get a functional jacket.
Stock-keeping
We would have a stock of Profitex Jackets, but electronic devices would be stored beside. An integration would be done right before delivering. Electronic parts have to be stocked in an adequate way.
Guarantee
It's also necessary to define the period of guarantee of the Profitex Jacket due to every single electronic part which is integrated.
Additional services
Cleaning service
An important issue to discuss is the washability of the jacket. Integrated electronic parts, which are not resistant against water, would have to be removed beforehand and be integrated again by trained staff. By entering through the zip you have to be able to reach the textile buttons which are located in both sleeves. Regarding this issue, we designed all bags in a way they can be opened easily.
Repair center
A repair centre, especially for the Profitex Jacket, would be necessary. People who can determine damages of the garment and of the electronic devices too, would be necessary. In a professional way, like within the cleaning process, the electronic parts have to be removed and integrated again. That would mean the responsible persons can handle all electronic devices - or send it back to the suppliers.
Fitting with Profitex jackets
After placing an order at Texport, we normally organise a fitting with the fire brigade. We would produce several sizes of the Profitex jacket and the firemen can try them on to find their perfect size.
Work Package 4 - Verification
Objectives
Within this work package the whole testing operations and activities of the project will be hosted.
Starting from defining the conditions, the single components, the lab examples and the system will be tested.
Description of Work
Task T 4.1 Definition of test standards
Within T 4.1 test standards for evaluating the whole project will be chosen and / or defined.
Results achieved in Task
With the development of this task is fulfilled the initial objectives where identified in a series of standard and nonstandard tests for the characterization of materials, components and systems that form part of Profitex Personal Protective Equipment.
Description of Work
Task T 4.2 Testing of single components
Each component manufactured in T2.2 will be tested regarding technical, safety, functional and physiological aspects. The standards defined in T 4.1 and listed in D 4.1 will be used.
Results achieved in Task
Task T 4.3 Testing of the lab examples
Based on the outcome of D 4.2 as well as D 1.1-D 1.4 single components were chosen to build lab examples. Within this task the lab examples will be tested and evaluated, also according the defined standards from D 4.1.
Results achieved in Task
Following table lists the elements that systems have been tested and their comments thereon. It lists the components that are integrated into firefighter's jacket, and peripherals that are part of the system.
Description of Work
Task T 4.4 Testing of the system
As a next step the system as a whole will be tested in regard to the overall functionality. The test standards from D4.1 will be used.
As result of T4.4 a report of all test results will be given (D4.5).
Results achieved in Task
According to D4.4 the current version of the system got a positive feedback by the involved fire fighters. Furthermore, the lab-based tests (D 4.2-4.3) show that the textiles and methods used for making the clothing are suitable for the application of ProFiTex.
Description of Work
Task T 4.5 Evaluation
All results of WP4 gained up to this point (D4.2 D4.3 D4.4) are evaluated against the user and system requirements (D1.3) as well as the test standards (D4.1). The overall performance of the system will be reviewed this way to make sure that the system fulfils all requirements and will not be an obstacle but a support for each firefighter using it. If necessary, last system adjustments might be carried out.
Results achieved in Task
The results of this task were reported together with the results of Task 4.4. That was regarding the scheduled testing session within the mid of the project. The final testing is scheduled at end of September, 2012. Accordingly, the evaluation results will be reported.
Work Package 5 Certification
Objectives
Testing, evaluation and preparation for certification of the system
Description of Work
Task T 5.1 Testing and evaluation under realistic conditions
The complete system will be tested in an exercise mission under controlled, but realistic conditions in the training center. Professional firefighters as well as volunteers will use the equipment. All relevant data will be recorded and the performance monitored.
Results achieved in Task
D4.7 and D5.2 have joined in the D5.3 with given the similar characteristics of these two tasks have decided to incorporate collect activities into a single deliverable.
The main results obtained under this task are:
A map of "Certifiable" parts of Profitex System, and the main features of this certification. Have been identified the elements that are subject to certification and what not. For this three components family has established: 1. Garment integrated components, 2. Peripheral components created in the project, 3. Commercial peripherals used in Profitex System.
- A certificate of the system parts that are in final version (or final working prototype) state.
- Certification recommendations for Profitex system parts that are in "non-final version". Will feature a series of protocols for improvement functional components (or devices) that are not in a final version. Will consider recommendations for future certification.
Profitex project proposal the following way in four steps:
- Identify modules of technology integrated on garments and its components
- Identify which components have a individual CE as market product.
- Certification of components without CE
- Final Module certification and CE marking.
Potential Impact:
Scientific articles and Press releases
General introduction articles to the project carried out for publication in mass media, presenting the scientific objectives of the project to raise interest in the scientific community, and periodic publication of the progress of the project
Brochures
In order to complement the dissemination activities for the purpose of promoting the exploitation of research results with activities aimed at a non-specialised general audience and to provide information on the nature of the activities carried out during the project and the benefits to the industrial sector, the partners elaborated general information such as flyers or posters presented in conferences and exhibitions.
Website
The project web site including informations about the project, objectives and expected out-comes, is been regularly updated. The Profitex website was developed and furnished with essential informations (e.g. project description) from the very beginning. The Profitex Home-page provides continuous project information such as: Introductory information, activities, partnerships, events
Geographical Coverage
The geographical focus of the dissemination activities within the time frame of Profitex is been carried out both at national and international level:
- At a national level partners, used their existing industry contacts to disseminate Profitex and promote the take-up of the developed technology.
- At International level, Profitex is been promoted in conferences providing contacts with relevant sectorial organizations.
The IPR Tool has been organized into two main sections:
- Learning section
- Innovation Management Section
Project website: http://www.project-profitex.eu
The aim of project ProFiTex was to support fire fighters in their perilous work with a system that supplies mission-relevant information without overwhelming the fire fighter. The system comprises two main components:
1. A fire fighting jacket with integrated electronics and a braided security rope, called Smart LifeLine which is able of transporting data. The jacket comprises electronic devices like an infrared camera, movement and localisation sensors and a volcano button integrated into the jacket sleeve. Wire-bound data transfer are be used since wireless communication is difficult over long distances and through several walls of a building. A security rope carried by fire fighters during a mission shall be equipped with data transmission capabilities. This allows information to be sent outside to the command post and back to the fire fighter. By monitoring several parameters of the fire fighters condition like his movement pattern and stance, problems can be detected immediately. The fire fighter himself is supplied with the possibility to navigate even in smoky environments thanks to the infrared camera and the positioning system implemented into his equipment.
2. Information will be displayed to the fire fighters, their group leaders and the commander outside the building. The amount and type of information supplied are being carefully chosen, considering the physical danger and psychic stress fire fighters are opposed to. Work safety of fire fighters shall be increased, thus lowering the number of work-related accidents and casualties. Fire fighting missions will be more efficient using the system.
Project Context and Objectives:
Fire fighting is a perilous job which brings people and equipment to the limits. Each year over 5.000 deaths are caused by fires in Europe alone, and among these dead are more than hundred firefighters. Providing technology for the professionals who fight these fires is the mission of the EU-funded project Profitex (http://www.project-profitex.eu).
Usability and user-centered design are a key point in the project. There are different roles and requirements in a fire-fighting intervention:
- The firefighter himself inside the building;
- The group leader, monitoring several teams of firefighters in the building;
- The mission commander, located outside the building, coordinating the intervention.
Project Results:
Description of Work
Task T 1.1 - Workplace Studies
In T 1.1 the consortium will conduct empirical studies of the working conditions of firefighters as relevant for the project's RTD objectives. The focus of these studies is on the actual characteristics of fire fighting operations but there are also other relevant aspects such as professional training, assessment of new technologies and adaptation of operational procedures within fire fighting organisations. The objective of these studies is to establish an empirically grounded understanding of:
- The operational constraints for the systems to be developed (the what)
- The constraints for an effective participation of the available firefighters in the design process and for the effective appropriation of the developed technologies into operational use (the how)
Results achieved in WP1
PROFITEX adopted the following focused objectives regarding workplace studies:
1) validate existing results and adapt where necessary;
2) create shared understanding among PROFITEX partners on firefighting work practice;
3) conduct complementary workplace studies where this provides substantial benefit.
This exercise was covered by the German TV channel 3SAT for their science program hitec: http://www.3sat.de/page/?source=/hitec/158804/index.html
Description of Work
Task T 1.2 - User Requirements
In T 1.2 will elicit user requirements focusing on the work situations identified and described in D 1.1. These user requirements are distinct from system requirements in that they do not imply any particular solution but simply state the problem or challenge for which the users, here the firefighters, require a solution. This task represents an important analytical effort to understand the actual needs of fire fighting operations in order to guide the search for and the assessment of possible solutions without focusing on any particular solution too early. Of course, the limited resources present in projects such as Profitex require focusing efforts on one or few solutions eventually. For the user requirements, the workplace scenarios described in D 1.1 as well as the project's technological capabilities described in D 2.1 will help in focusing our efforts.
T 1.2 will report the user requirements in D 1.3.
Results achieved in Task
These requirements were written with respect to Volere Requirement standards. This requirement standard distinguishes itself from other requirements standards by testing the requirements while they are being written. This is carried out by looking associating a fit criterion with each requirement. This fit criterion is used to determine if the proposed solution can fit with this requirement or not.
Description of Work
Task T 1.3 - Prototyping and Participatory Design
In T 1.3 the consortium will engage firefighters and other stakeholders in a participatory design process. This activity is based on the understanding that usable and eventually accepted designs do not result from the implementation of specifications that have been obtained before the design phase. Instead, successful designs result from many iterations of putting the artefacts to work in the hands of the users and obtaining their feedback. To enable this process even before functional system prototypes become available, T 1.3 will prepare and organize a series of workshops where the use of systems or system components during fire fighting operations will be simulated with different means, based on the prototyping approaches developed in the wearIT@work project. In particular this includes simulating fire fighting operations with the FireSim prototyping tools, allowing multiple actors to use virtually simulated future technologies in either a fully virtual or a mixed-reality environment, i.e. an augmented physical training facility or a virtual building. The purpose of these activities is providing to the firefighters as rich as possible an experience of using future technologies in realistic usage situations and thus enabling them to provide design-relevant feedback in a meaningful way.
Results achieved in Task
In addition to a continued improvement of the FireSim prototyping platform, from version 1.3 at the midterm review to version 1.6 at the final review, with virtually hundreds of bug fixes and improvements, two major additions were made during this time. First, the Command Post System prototype was implemented as part of FireSim, enabling interactive testing with very limited effort and very shortly after the concept had been defined. This implementation was also used in the studies reported as part of T1.1.
Description of Work
Task T 1.4 - System Requirements
In T 1.4 the specific requirements for a system addressing a defined set of user requirements from D 1.3 will be described. The first step here is to select a coherent subset of requirements from D 1.3 that Profitex will address. This selection will depend on the project's feasibility evaluation regarding different technological options in WP 3 and a prioritization of requirements by the firefighters. Similar to the user requirements, the system requirements will not specify a concrete implementation but provide measurable criteria for judging whether any given system meets the requirements.
Results achieved in Task
Task T 1.5 - Economic analysis and financial planning
With the aim to develop an economic analysis which supports assumptions made on demands in the short- and medium-term, TexClubTec will provide market data and data of potential customers and fire brigades.
Results achieved in Task
Within this task potential customers of Profitex were identified. Beside fire brigades TexClubTec (TCT) identified army, police, national and regional civil defence organizations and industrial companies with own internal fire fighter services. The army and police need a different configuration of e.g. sensors to be used in their application scenarios. Search & rescue scenarios in unknown and dangerous environments can be transferred immediately. Also the concept of the information centre - the command post - is very useful for the other target groups. While the main focus of a special force of the police or army is not fighting fire but rescuing people out of dangerous situations and neutralizing one or more suspects in not predictable areas, the Profitex system enhances army and police operations due to the advanced information technology implementation.
Description of Work
Task T 1.6 - User Reference Group: Workshops and Online Community
T1.6 deals specifically with organizing an exchange of knowledge and experience among the members of the User Reference Group on topics relevant to the Profitex project. While the aim of T1.3 is to engage end-users in hands-on design activities one organization at a time, T1.6 focuses on bringing the different perspectives from end-users together by fostering an open debate among the members of the User Reference Group. This exchange will be facilitated by two principal means: 1) dedicated annual workshops where representatives from all end-user partners as well as the technical partners will work on principal aspects of the project and 2) an online community where a continuous exchange is motivated and facilitated, focusing more on a sustainable lively discussion than on the exchange of abstract information.
Results achieved in Task
The realization within PROFITEX of not being able to develop functional physical LifeNet beacons let to a re-evaluation of the Reference User Group (RUG) strategy. As explained above, the inability to develop LifeNet beacons within PROFITEX required the search for an alternative localization technology and the adaptation of the navigation support system to this technology. To this end, a more targeted technical discussion was required which was only practical within a smaller group of end-users and domain experts familiar with challenges faced and available to engage in a sustained technical discussion. Therefore, this discussion was conducted with fire fighters from HEAT and other domain experts from within and external to the PROFITEX project.
Work Package 2 - Technology Development
Objectives
Within this work package the planning and design of the single components will be done. Starting from a definition the components will be planned. Furthermore the components will be manufactured, and in combination with WP 4 they will be tested. In the end a decision about which components will be used for the next steps will be made. This will finally lead to a system design, that can be tested under real conditions.
Description of Work
Task T 2.1 Definition and planning of single components
The following technical parts are essential for the whole concept.
Power Supply System
- Definition of a modular component system
- Evaluation of the data volume
- Definition of internal (jacket based ) interfaces (Textile Bus System)
- Definition of the jacket to lifeline interface
- Definition of buttons and display (HID)
- Definition of system behaviour in case of system failure
- Security aspects ('is there any danger coming from the system?')
- Definition of performance
- Results achieved in Task
Requirement plan
Protective clothing is closely related to the clothing sector. Potential solutions and product designs are complex due to their intended purposes. Considering protective clothing for firefighters there is mainly the task of heat protection and the transmission of sweat. To fulfill all these tasks standard fire-protective clothing consists of several layers. The layer system depends on the specific model of a garment.
Basically it is structured into
- Outer shell
- GORE-TEX membrane
- Inner liner
Requirement plan
In order not to endanger firefighters, protective clothing must meet the following requirements.
- Thermal Protection
- Moisture and hot steam barriers
- Re-drying and sweat permeability
- Weight and ergonomics
- Visibility
- Robustness
- Haptic
Description of Work
In this task all components are manufactured by the technology partners. If necessary, not only one technical layout will be manufactured, in certain cases also two or even three different approaches might be necessary.
Results achieved in Task
Smart Lifeline Development
The development of the Smart Lifeline was done in the course of a PhD thesis. All images and results describing the Smart Lifeline development are taken from [Eic12] with the author's permission.
Conventional data cables
In a first step regular data cables have been analysed and their main components were identified. An electromagnetic shielding (in this case a metal-coated polymer foil) is wrapped around the cable. The outer layer is formed of an extruded polymer jacket.
Smart Lifeline Concepts
The Smart Lifeline has been developed in two variants. Variant A consists of a textile data cable with no electronic devices in between the two ends of the cable. The main goal in the design and production of this data cable is to research the integration of data lines in braided structures. Critical parameters like cable resistance and signal transmission quality are to be assessed with this prototype.
The components of the Variant A prototype are as follows:
- Core rope (for mechanical stability)
- Functional layer (holds the electric leads
- Insulation layer (separates the functional layer and the EMI shielding layer)
- EMI shielding layer (against electromagnetic disturbances)
- Mantle layer (mechanical protection).
The elements in this design are as follows:
- Core rope (for mechanical strength)
- Woven narrow fabric with electrical leads
- Adaptor, bottom shell
- Adaptor, top shell
- Electrode channels (here the electrodes of the electronic device can be plugged in)
- Insulating braiding layer
- EMI shielding layer
- Mantle layer
- Electronic device
Measurements procedures
After producing the Smart Lifeline prototypes each one was tested considering its signal transmission capabilities. The Smart Lifeline Variant A and B were subjected to the the following tests:
- Laboratory tests (Variant A and B prototypes)
- Insulation test (testing against short-circuits)
- Ohmic resistance measurement (DC resistance)
- Impedance measurement (AC resistance)
- Signal attenuation (signal loss from one end to the other)
- Cross-Talk
- Practical tests
- USB Data transmission test (variant A only)
- Video signal transmission test (variant A and B)
- displays the equipment which was employed during the tests.
Insulation test
Two variants of the Variant A prototype were produced. The electrical leads in the functional layer were realized using four Shieldex yarn (235 dtex) in one variant and copper wire (d = 0.15 mm) in the other variant. An insulation test was performed. This was done by testing the connectivity between 1A-1B-1C-1D-1M. Only 1A is supposed to show a connection. If any other pairing shows an electrical connect, a short-circuit has occurred. Next 2A, 2B, 2C, 2D, 2M are tested and so on.
Ohmic resistance measurement
Cross-talk
Cross-talk is a phenomenon which occurs between neighbouring electrical leads. A signal flowing through a conductor induces a signal in the neighbouring lead, resulting in a signal loss in the signal line and noise in the passive line.
Practical test 1: USB data transmission test
To show the capability of the braided data cable (variant A) USB connectors have been soldered to the cable. USB contains four data leads and a mass lead, same as the variant A prototypes. Therefore variant A prototypes have been confectioned with USB type A and USB type B plugs to connect a notebook and an external harddrive.
Practical test 2: Video signal transmission test
With both the variant A and B prototypes of the Smart Lifeline a practical video transmission test was conducted. The signal was sent from a source (DVD player) to a TV by using an analog composite video signal feed. Composite video requires only two eletrical leads - one for the signal and one for mass. The variant A prototype was confectioned with cinch connectors. The four data leads were all soldered to the signal pin of the cinch connector, the EMI shielding layer was connected to the mass.
Insulation test
The insulation test was performed on both the variant A prototypes (braided data cable with wither copper wire or silver-coated Polyamide leads). The prototype with copper wires exhibited no short-circuits, all four leads and the EMI shielding are clearly separated.
Ohmic resistance, Impedance, Signal attenuation, Cross-talk
For comparison all measurements have also been performed on a standard Cat-5 Ethernet cable. Since signal attenuation and crosstalk are effects which depend on the signal frequency, these measurements were taken at 100 kHz, 1 MHz, 10 MHz and 50 MHz each.
Practical test 1: USB data transmission test
Two variant A prototypes were outfitted with USB type A and B plugs, one variant A prototype was confectioned with USB type A-A plugs.
Practical test 2: Video signal transmission test
The images from the source (DVD player) could be transmitted clearly, there were no disturbances in the video image. This demonstrates that the textile data cables are capable of transmitting a clear video feed.
Beacon Demonstrator
Extensive research on local indoor positioning and localization has been done at Labor to verify the possibility to integrate a powerful computational unit into the SLL beacons and able to provide global and/or local position information with high resolution and accuracy.
Hardware
The development platform previously selected for the Beacons development was dropped once the localization requirement was taken out from beacons specifications. A more simplified architecture and hardware was then chosen.
Firmware
The firmware flashed into the Beacon is able to calculate the retreat path along a series of beacons using an algorithm based on Dijkstra calculation. It also sends and receives messages to the Host coordinator, the TMS1 in the Display, and records the positions of all the other beacons deployed in the GUI software. It also drives the two Green and Red LEDs to signal the Fire Fighter proximity (Red Led) and the route by flashing the Green Led. There is also a local temperature sensor constantly monitored by the CPU; once the temperature is above a preset threshold, the beacon is automatically excluded by the retreat path and each Beacon modifies its routing map accordingly. The routing map is also updated every time a fire and/or a broken path is active in the scenario. All the communications are managed by the ZigBee Module and coordinated by the CPU.
GUI Software PC
The GUI interface is designed to simulate the deployment of a number of beacons in a SaR mission and test the real beacons behaviour when different dangerous situations take place like the presence of a fire, a connection path between two beacons is broken or when a beacon senses a high temperature over a certain threshold.
Fire Fighting Jacket Development
The main goal is to investigate a textile based interface for the fire fighters' protective clothing. It can be used for communication between fire fighter and corresponding system enabling to transmit data to both the fire fighter inside the building and the group leader outside the building. The fire fighters' garment is the medium for integrating the interface and electronics to build up an infrastructure to distribute data and energy. In that way, it will be possible to locate the fire fighter. The data will also be used to monitor the posture and activity of the fire fighter (stance recognition, activity index).
Plan for the Textile Keypad
When combining the HID with a standard fire fighters' garment it does not mean to destroy the garment's functionality, but to full fill the exact statutory provisions and norms. For instance to reach the thermal protection of the fire fighters exposed to 1000°C and to maintain a constant body temperature heat regulation mechanisms with re-dryable and sweat permeable materials.
Structure of the HID
The HID is realized in three devices which are placed at the forearm at the fire fighter garment. Every device is composed of the following:
- water resistant cover protecting the conductive materials against influences like extinguishing water from outside
- relatively broad and wide knitted spacer fabric to get electronic contact by pressing the device (even wearing thick fire fighters` gloves)
- button matrix with the opportunity of nine contact points
The following technology is used for building up the textile matrix switch. The HID consists of three layers.
Water resistant cover:
To protect the electronic components from outside influences there is an upper functional layer. This water-resistant cover is made out of waxed awnings to avoid influence on the from outside such as water and heat. To produce it there are two waxed awnings sewed together. One of them forms a quadratic button whereas the other part forms the ground.
Spacer Fabric:
This functional layer is followed by a spacer fabric to ensure a high material flexibility of three-dimensional feature: spacer fabric which is manufactured by Müller Textil which has lot of experience in the textile warp knitting technology. This fabric is 10 mm thick.
Button matrix:
The third layer consists of a button matrix including silvercoated polyamide (PA) yarns on the inside of the buttons: button matrix. The button matrix consists of two fabrics. By rotating the bottom layer around 90 degrees, a matrix can be formed. The bottom layer has stripes of conductive yarns that connect through a tape with wires (special narrow fabric from OFFRAY®) with the USB-connector. A housing of the electronic interface is needed to encapsulate the electronic and the cabling.
The button matrix is structured like following:
- Upper Function layer
- Knitted fabrics
- Knitted fabric with several stripes of conductive yarns
- Bottom Function layer
Process
step Tools which are needed Main Process Textiles which are needed pictogram
1 Scissors Cutting 2 Knitted Fabrics
- Non-woven
- Fabric with several stripes of conductive yarns
2 Hot iron (140°C) Bonding Fabric
- Non-woven
- Fabric
- Knitted fabric with several stripes of conductive yarns
- Unformed Upper function layer
3 Former plate, Punching tools Thermoforming Unformed Upper function layer
Upper function layer
- Bottom function layer
The bottom function layer consists of the following three fabrics containing diverse properties:
- Woven fabric with several stripes of conductive yarns
Several paths are tightly woven with silver coated conductive yarns into a textile, produced with PES-yarn on the web loom.
- Supporter fabric
The upper function layer can be attached to the supporter fabric to guarantee the required stability for the keypad.
- Awning fabric
Apart from being a supporter fabric this awning fabric also offers additional textile stiffness for the keypad and a hard ground.
Position of the human-interface device inside the fire fighter jacket
Generally the layer system depends on the model of the jacket. The layer system is structured like following:
1.) Shell
2.) GORE-TEX membrane
GORE-TEX membrane is used to guarantee the waterproof despite it is breathable so that it prevents the fire fighters against wind and super heated steam.
3.) Interlining
The HID in form of an additional layer should be placed between the GORE-TEX membrane and the interlining. This way the HID is protected against influences like extinguishing water and rain as well as the high temperatures from outside.
Materials and methods
One of three layers is a knitted fabric with several stripes of silver coated conductive PES-yarn inside. To get the form of the button matrix the knitted fabric including the conductive stripes has been heat set with a heated press machine like mentioned above.
The test apparatus has the following characteristics:
The tactual finger is a steel pin with a diameter of 10 mm coated with foam which is driven by an electrical motor and operated with a sinusoidal output.
Parameters Technical details
Cycle sinusoidal form
Tactual finger Steel pin with coating
Underlay Foam PUR F46
Switching voltage 5 Voltage
Measuring frequency 200 Hz
Pulse frequency 0,7 Hz
material Button matrix
Results
Every button has been tested on its pressing stability for more than 80.000 switching cycles and have been finished manually. It should be noted that the thermally formed button being tested has to retain its switching function over the testing time. But the surface and material stability of the buttons become severely affected after this high switching numbers which has an effect on sagging of the thermally formed three layered knitted fabrics.
The aim of both tests was to provide a relation between human sensing and the influencing parameter
- Thickness of surface material (number of layers)
- Height of transparency curvature (mm): The height of transparency curvature makes the difference between a soft and hard switch.
The haptic tests have been carried out as a questionnaire. The results of the haptic tests are related to the pressing force tests.
- 2- and 3-layered surface with high transparency curvature (> 4,3 mm) possess a pressure point Surface created with more than 3 layers are too stiff
- Low transparency curvatures (< 3,6 mm) do not achieve enough counter pressure
- Sensing of a tactile feedback with the Transparency Textile Compound Switch in contrast to the Textile Switch.
Conclusion
Although the HID worked well under defined Lab Conditions, the function of the buttons gets worse and also the tactile feedback of the user was poor in real user tests. Like we set up before the major criteria is to have a tactile robust and serious feedback and a rugged electrical contact. It is essential that the fire fighter is able to feel a feedback coming from the button. This textile buttons worked well under defined Lab Conditions, but while using them in simulations they turned out to be not yet reliable. Due to the fire gloves this was a serious problem. Also the ruggedness was too weak. The major criteria were to have a tactile a functional button in the meaning of having a rugged electrical contact and also a robust and serious user feedback.
Manufacturing of single components (volcano button)
Due to the experience of the tests we decided to develop a new version of buttons. It is essential that the firefighter is able to feel a feedback coming from the buttons. We also tried to create a feedback loop using symbols in the display, but this was not usable under fire fighting conditions. Also a acoustical feedback, (as known from mobile phones) is not an option due to the noise in fire fighting situations. Most important issue was to find a design that avoids unwanted release. So the main issue of this design was to combine the tactile feedback and a kind of protection boarder to ensure that the user will not activate the button unwanted. This is very important because this would affect the systems performance dramatically. The lead to a design we call 'Volcano Button'. The name is coming from the shape of a volcano, which is pretty the same shape as we realized for the buttons. The Button is a assembly of three parts.
The following drawing shows the construction of this assembly.
It was very important to find a design that fits to the needs of fire fighting applications. This means that we have to find the correct dimensions for the volcano button design. We evaluated a lot of different fire fighting gloves and this leads us to a button dimension of approximately 30mm. The first thing was to find a water tight button (IP67) witch have to be electrically solid and also heat resistance.
This finally results in the following parts:
The next important thing to connect these buttons to the FFS unit was to find an Interface. Therefore it was necessary to convert the bush button signals to USB commands. We used a special electronic for doing this. It is a electronic called AVR Stick.
Description of Work
Task T 2.3 Integration into lab examples
The single components are integrated into textiles. The choice will be made based on the results of the testing in T4.2 The textile material is equal to the material of the end product. Electronic components are connected with textile data energy buses to demonstrate and verify the data and energy transmission.
Results achieved in Task
The main focus of the 'Profitex Firefighting Jacket' will be the practical functionality of the jacket. A general textile integration of the electronic device is possible.
Material composition
The 'Fire Breaker Action Jacket' contains a liner system with the following parts: antistatic outer Shell, Gore-Tex Membrane and Lining. 'Liner system' means that the membrane is placed loose inverted between outershell and inner-lining. It's not laminated with the outershell. This is a big advantage for having the best and easiest possibilities in integrating all needed components.
Gore-Tex Airlock Membrane:
Water tightness and high thermal protection are given due to the intact and breathable Gore-Tex Airlock membrane. The membrane also offers advantages like high protection against contaminated liquids and also high reduction of weight.
Manufacturing
Integration of the necessary technological components into the manufacturing process is possible. Always one component gets modified or changed, we have to adapt the steps in manufacturing. Main important for all developments is of course the functionality in practice. An important issue to discuss is the washability of the jacket. Integrated electronic parts, which are not resistant against water, have to be removed beforehand and integrated again by trained staff. This means additional work and expense, also additional service costs. Efficient solutions have to be found. Also the wearabilty of the Profitex Jacket has to be evaluated in practical experience.
Task T 2.4 Evaluation and drafting of a textile integration standard
The components are evaluated considering their mechanical properties. The compound textile-electronic component is characterised. Compiling the experiences of the textile integration, a standard for the integration of electronic equipment into fire-protection textiles will be drafted. Considering the future production of the Profitex system, production methods and production costs will be evaluated in this step as well.
Deliverables will be:
D2.8 Report on the evaluation of the system design
D2.9 Standard for the integration of textiles into fire-protection components
Results achieved in Task
A certification for the Profitex jacket has not been done so far, as this certification would also include the certification for the electronic integrated devices and components.
Description of Work
Task T 2.5 Signal-processing for components
VUT and ETH have experience and ongoing research projects in pattern recognition and signal processing to interprete and analyze data from bio-sensors.
Results achieved in Task
Bio-sensing
For an 'everyday life and field application', a minimal sensor setup is desired for comfort reasons.
The monitoring belt consists of three smart fabric sensors to acquire cardiac activity, breathing rate and skin temperature (Zephyr). The body acceleration is measured with a 3D accelerometer included in the recording device. In addition to ECG data, the chest belt provides RR intervals by measuring the duration between two consecutive R waves of the ECG.
Work Package 3 - System Development
Objectives
Creation of a complete textile system
Description of Work
Task T 3.1 Firefighter System
In Task 3.1 the specification, design, development and integration of all components belonging to the system used by the operational firefighters is carried out. This task includes the creation of early physical prototypes to investigate ergonomic options. The firefighter system also includes all augmented tools that might be carried by the firefighters such as the Smart LifeLine.
Results achieved in Task
Fire Fighter System
Computing Unit
Following the results presented in the Midterm report and the decision taken at the Workshop held in Rome in October 2011, a new computing unit for FFS2 has been developed to better accommodate Firefighters needs. The new computing unit has been dramatically reduced in its dimensions, weight and power consumption. The operating system has been switched from Windows to Android. All heat and thermal issues have been washed away with the new implementation.
Hardware
After some research Labor targeted the main board to be the versatile BeagleBoard-xM (BB-xM), a Laptop- like performance system based on a low power TI ARM CORTEX A8 with high graphical and computational capabilities.
The BB-xM has been physically modified and integrated with extra custom circuitry, electronics and hardware for battery management, protection and integration with the Computing Unit Case.
The case was designed to be IP68 compliant with rubber rings and perimetral screws on the top cover to guarantee the appropriate sealing.
3D Reconstruction & Tracking and Thermal Mapping
Tracking the location of firefighters in fire brigade operations is one of the main objectives of the Profitex project. In addition to their position in space, it is important for an operational commander to know about the structural context (i.e. building structure) in order to safely guide his men and women during an operation. 3d models or accurate maps of buildings are rarely available before an operation. Therefore, our system has been designed to reconstruct the 3D structure of a building in real-time. The constructed mesh can then be used to provide an operational commander with a better overview of the situation. Simultaneously with the reconstruction of the 3D structure, a fire fighters position and viewing direction is calculated relative to it.
Thermal Imaging
Thermal imaging is important for a firefighter's security in his or her daily work. Firefighters use thermal cameras to see in dark and smoky environments, to search for victims and to find hot-spots and other potential dangers. From literature the importance of thermal imaging for firefighting becomes obvious, for example in (Amon et al. 2005), where a detailed report of a workshop on thermal imaging research needs for first responders is given. Therefore, we have integrated thermal imaging in our live system to enhance informative value and expressiveness of the 3D reconstruction.
SLAM
State of the art approaches give a solution to the problem of self-localisation and reconstruction in unknown environments by realizing Simultaneous Localisation and Mapping (SLAM) with visual data and additional sensors. SLAM has initially been developed in the robotics community and has undergone great advances in the last decade (Bailey & Durrant-Whyte 2006)(Reitmayr et al. 2010). The aim is to simultaneously build a map in an unknown environment and to calculate the objects position within this map, which is very useful if there is no map available. The main approaches to solve SLAM are EKF-SLAM (R. Smith et al. 1990), Fast-SLAM (Montemerlo et al. 2002), and Vision-Based-SLAM (Sim et al. 2005) among others. These approaches rely on information obtained from sensors such as sonar, stereo cameras, laser scanner, etc., and/or odometry. Most of them have been applied for robots, unmanned vehicles or aircrafts. Furthermore, MonoSLAM and the more accurate Parallel Tracking and Mapping (PTAM) system (Klein & Murray 2007) allow to investigate flexible environments with 3D infrastructure.
Depth Cameras and Kinect Fusion
Depth cameras provide input to another class of mapping and tracking algorithms e.g. (R. A. Newcombe et al. 2011). Here, the depth information is not calculated from 2D images, but directly streamed from a sensor.
Thermal Cameras
Cameras that generate thermal images in the long wave infrared range (LWIR) are referred to as thermal cameras or FLIR (forward-looking infrared) cameras. Thermal cameras, operate in the long wave infrared band which refers to radiation with wavelengths in the range of 8-15µm. Each object (surface) with a temperature higher than absolute zero (0K) emits energy with a certain wavelength. The hotter the object the more energy is emitted. The LWIR band captures most of the wavelengths of thermal radiation that we are confronted with in our natural environment. Thermal cameras are able to measure the emitted infrared radiation in the LWIR band and visualize these thermal patterns as an image (Balaras & Argiriou 2002).
Hardware & Setup
In our system we are using an Asus Xtion Pro as depth sensor, which provides depth data from the structured light approach described in subsection 0. The depth data is streamed at a resolution of 640x480 pixel and a frame rate of 30 frames per second.
Operating System
The Operating System installed on the Beagle Board and chosen by the consortium was Android. The appropriate Android OS kernel version has been tested and crafted on the BB-xM to interface the Firefighter System devices:
- Helmet Display
- HMI
- Breadcrumb Wireless communication device
- Bluetooth Health system monitor and PDR
Two versions were tested before choosing the final one: Gingerbread 2.3.4 and IceCreamSandwich 4.0. The final OS installed on the CU for the FFS2 is the Ice CreamSandwich 4.0 which proved to be more stable and faster than the GingerBread version.
Breadcrumb
The Breadcrumb device integration was conditioned to have a separate source battery supply and mounted on the Fire Fighter Oxygen tank. The first proposal sketch is reported below.
The subsequent change to this sketch was the cable arrangement: instead of having a secondary plug on the back of the jacket, it was decided to reuse the SLL connector from FFS1 and run the breadcrumb cable, together with the Helmet-Display cable, along the shoulder strap down to the connector.
Description of Work
Task T 3.2 Group Leader System
In Task 3.2 the specification, design, development and integration of all components belonging to the system used by the group leader is carried out. This system supports the group leader in receiving information from the team for monitoring and planning purposes, in detecting critical conditions, and in taking decisions to secure the safe and efficient evolution of the operation. For example, this system is supposed to provide information on the search status of a given building floor based on the localized information provided through the Smart LifeLine. It is important to note that the development of a completely new hardware device for the Group Leader System is not within the scope of the Profitex project. Instead, the consortium plans to use and integrate available hardware such as a ruggedized TabletPC or UltramobilePC.
Results achieved in Task
Design and Implementation of the 3D-Reconstruction and Thermal Mapping
Kinect Fusion Algorithm
The Kinect Fusion algorithm (R. A. Newcombe et al. 2011) processes every frame obtained from the depth camera and generates two things: an updated 3D structure containing information on surfaces in the scene and the camera pose.
In order to do this, multiple steps have to be executed:
- Preprocessing: From the current 2.5D image, positions and their normals are calculated at multiple resolutions.
- Pose Estimation: Using the Iterative Closest Point (ICP) method the pose of the current frame is aligned with the previous surface data stored in single combined 3D structure - the volumetric truncated signed distance function (TSDF) (Curless & Levoy 1996). In fact, the frame is not aligned with the whole data structure, but only that part of it, where the prediction of step 4 supposes that the frame should be located. Therefore, the fast projective data association algorithm (Blais & Levine 1995) can be used.
- Reconstruction Update: Once aligned, information contained in the current depth frame is merged in the TSDF volume. Over multiple frames the depth/surface values in the TSDF structure are averaged. Therefore, the final TSDF structure contains less noise and holes and is much more accurate than a single depth image.
- Surface Prediction: Based on the last poses of the camera the next pose is estimated and using ray-casting a surface model from that perspective is created from the TSDF volume. This local surface model is then used in step 2 to align the next frame.
From the TSDF volume a point cloud, mesh or other representation of the surfaces in the scene can be finally extracted.
Implementation Kinect Fusion
Our implementation of the Kinect Fusion algorithm has been developed in C++ and is based on the Point Cloud Library (PCL) in large parts. We make use of the available methods implemented in CUDA and optimized for processing on the GPU.
Description of Work
Task T 3.3 Command Post System
In Task 3.3 the consortium will address, with a relatively small effort, the question of how the information obtained through the Profitex systems may be made available and exploitable at an information system at the command post. Similar to the group leader but one or two levels (in case there are sector chiefs) up the hierarchy, the command post is in charge of keeping track of the engaged teams and their missions. Generally today, command posts are not concerned with information on individual firefighters. Nonetheless, support for handling aggregated information is likely to be of interest. And as before, special circumstances such as lost contact to an unresponsive firefighter may warrant focusing the command post's attention on an individual.
Results achieved in Task
The Command Post System in Profitex was realized using COTS devices.
The approach is to provide operational pictures with schematic representations of tactically relevant information (connectivity, tagged locations, responders, hazards). Schematic representations are used to deal with the inaccuracies and sometimes inconsistencies of the available information and the fact that this information becomes available gradually over the course of an intervention.
Description of Work
Task T 3.4 Monitoring the firefighter's activity and health
This subtask is concerned with the implementation of procedures to continuously monitor the firefighter's activity and health during a mission. The resulting indicators of activity and health will be provided to the commander of the mission by using the communication infrastructure that is developed in WP2. In this way, the commander is always aware of the physical and physiological status of his firefighters and can react accordingly.
In order to achieve a continuous monitoring of the firefighter's activity and health, the following working steps will be carried out:
- Selection and adaptation of available sensor technology in cooperation with other technological partners
- Characterization of sensors in terms of signal quality, stability and liability to motion artefacts
- Design and implementation of laboratory experiments in order to build up a sensor data repository
- Sensor data pre-processing and feature computation
- Modelling of activity- and health-indicators based on selected signal features
Results achieved in Task
In the following we introduce the sensing equipment to monitor physiological health status, physical activity and cognitive performance.
Physiological Monitoring
In our experiments we used Zephyr first responder system BioHarness (Zephyr) as the physiological monitoring system as explained in Task 2.5. The Zephyr system is a mobile multi sensor chest belt. It combines physiological parameters with activity measurements and posture analysis. The sensor electronics module (SEM) measures heart rate (derived from electrocardiogram (ECG)), breathing rate, skin temperature, activity and posture data.
Activity Monitoring
For the activity monitoring we have used orientation sensors to capture the body motion and orientation of each fire fighter. The sensors were attached to the head, the torso, the lower arms and the right foot. This not only allows us to measure the activity level of each fire fighter in a fine grained manner for each body part, but also enables us to investigate the coordination behavior of two firefighters in the same troop.
Monitoring of Cognitive Performance
Cognitive assessment is the examination conducted to determine subject's level of cognitive function and skills. In this task cognitive assessment is investigated to identify mild cognitive decline in order to determine the ability to manage complex activities such as search and rescue. Several standardized desktop-based cognitive assessments have been designed in which users have to watch a display and respond actively in the test by using keyboard, mouse or special buttons. These tests measure functions such as reaction time, vigilance, attention, learning, memory and sensory function. Scientists use such tests to assess mental performance; however, they are not feasible to be used in the field. One limitation is that the user has to stop all ongoing activities to participate in testing and another is that the equipment is often not portable. Therefore we designed a wearable device which is able to measure reaction times of a person during activities of daily life in order to assess cognitive functioning.
Activity Level and Troop Coordination
When the troop receives their mission command they start with the search and rescue mission and their activity level increases rapidly (green and red colour). The diagonal trend from the lower left to the upper right depicts the coordination behaviour of the two fire fighters in the sense that the first firefighter is followed by the second fire fighter. It can be seen, that the coordination behaviour changes at the end of the mission when the person is being rescued because now the second firefighter leads the way out and the first firefighter follows.
Physical Workload
We divided the data into three segments (i) baseline (shown in green), (ii) search and rescue (shown in red), and (iii) recovery (shown in green). According to Zephyr data, the mean heart rate of the firefighter was 96 during the baseline period. High value of the baseline HR can be explained by the heavy firefighter gear. During search and rescue mission, the mean heart rate of the firefighter reached to 140 BPM and the mean HR decreased to 119 during the recovery period. The effect of the high physical workload can still be seen on the recovery period since it needs longer time to calm down.
Posture Mirroring
The automatic detection of nonverbal cues such as body movement from sensor data provides a way to capture aspects of human behavior. An important nonverbal named posture mirroring occurs when persons display similar postures while interacting with one another. Research in psychology has linked posture mirroring to increase rapport and empathy and to support communication. This supportive functioning could be important to monitor during fire fighting missions as it would allow commanders to gain an insight into the nonverbal communication patterns between fire fighters throughout a mission. This knowledge could then be used by the commander in post incident briefings and further fire fighter training.
Monitoring firefighting teams during training scenarios
We investigated the use of smart phones to monitor firefighting teams in order to improve post incident feedback and to provide better training. We have explored what basic team behaviors such as team activity and team communication can be measured with the smart phone in the context of firefighting. We rely on built-in sensors of the smartphone because firefighters are used to carry their mobile phone with them even during incidents.
Experiment
The experiment was conducted in the fire simulation building at the training facilities of the Zurich fire brigade. In the fire simulation building a variety of fire scenarios can be realistically simulated ranging from kitchen fires to a burning car in the garage. During training, firefighters are confronted with real fires, extreme heat, high humidity, severely restricted visibility and thick smoke.
We have recorded motion and audio data of 10 civil firefighters during one day of training. In the chosen scenario a kitchen fire in the third floor of the training building had to be extinguished. Two teams consisting of five firefighters each completed the scenario. After the first team finished the scenario, the instructor gave feedback to all firefighters and highlighted mistakes that should be improved, then the second team started with the same scenario.
Team communication estimation
In order to capture the amount of verbal communication during a firefighting incident, we employ the robust voice activity detection algorithm presented in [1]. At its core a long-term signal variability (LTSV) measure is used to measure the degree of non-stationarity in the audio signal and is hypothesized that speech has a higher degree of non-stationarity as compared to noise sources. Because the LTSV-measure is independent to amplitude scaling of the input signal it is robust and well suited for the harsh firefighting environment. We tested the detection accuracy by manually annotating speech of four firefighters during the training scenario and obtained detection accuracies between 75% and 85% even in high noise conditions.
Body and speech activity
Body and speech activity describes the fraction of a two second window that a firefighter was active. The body and speech activity level of the two teams is displayed. When comparing the two teams it becomes obvious that team 2 was faster than team 1. When comparing the body activity of the engineer it becomes clear that the engineer in team 2 was much more active than the engineer in team 1. Concentrating on the preparation phase we can infer that team 2 appears to be better coordinated as all members are first active and before they continue to work, first stop to communicate which can be seen by more speech activity between the troop leader and the incident commander. This pattern is notably absent in team 1.
Conclusion
We have demonstrated that the measurement of important team behaviors such as the amount of communication and the mount of body activity can be measured with the smartphone in a typical firefighting scenario. Moreover, we showed the potential how the measures could explain why one team was faster than another. In future research this needs to be validated with a larger sample size to proof generalizability.
Description of Work
Task T 3.5 Visualization of video data
Enhancing the video signal from the worn infrared camera is vital for the firefighter to make the correct decisions when confronted with dangers within his environment. The image data from the video signal is a grayscale presentation of the temperature distribution in the surrounding scenario. To assist the firefighters in a very stressful situation the grayscale values of the image can be mapped to different color schemes to give the firefighter a more meaningful picture of what is happening around him. These mapping techniques are common practise in volume visualization. In medical application different tissues of the human body from scanned CT or MR data are emphasized by so called transfer-functions. Thus here it is possible to enhance the grayscale video image by highlighting humans and emphasizing objects in the surroundings that exhibit a potential risk for the firefighters, like objects that exceed a specific temperature which are dangerous for them.
Results achieved in Task
First tests with our system in different situations showed good results. We were able to reconstruct different large scenes like a building entrance, different rooms, hallways and staircases with a level of detail, which should be more than sufficient for the proposed purpose.
Thermal imaging with calibrated temperatures enabled us to clearly highlight human body temperature in environments with room temperature. Our experiments with thermal imaging depicting relative temperatures suggest their usefulness even in situations, where the temperature differences are smaller.
Limitations and Future Work
We have not yet tested our system in dense smoke, but from the technologies used, it has to be expected that performance will be significantly hampered by anything blocking or degrading the quality of a direct view in the visual and especially in the near-infrared spectrum. In environments densely covered with smoke it is very likely that reconstruction is brought to a halt. In this case a user could, however, still revert to thermal imaging to retrieve information about structures and people in the building.
Description of Work
Task T 3.6 Integration
In Task 3.6 an integration plan is defined and executed to ensure that all partial systems developed in WP 3 can successfully form one interoperable system. Experience shows that this integration effort requires dedicated responsibility within RTD projects and that it needs to be addressed not after the components become available but ideally before development starts.
Task 3.6 will result in an Integration Plan (D 3.13).
Results achieved in Task
Integration Plan
All cables which are necessary for electronic devices are hidden and not visible for the wearer of the jacket. Only some cables, which belong to the CPU and go down to the bottom of the jacket, are visible.
Cables are fixed on the inside with a velcro loop to avoid moving.
Materials
In the beginning we had doubts regarding the implantation of the textile integration. Cables should go from the outside to the inside of the jacket, without damaging the membrane. Integrated electronic device should not be in contact with heat or water.
Production
Another difficulty is the integration of all electronic devices during manufacturing process; it's not possible to integrate it to the finished jacket. Our dressmaker in the manufacturing plants would have to work with new materials and accessories. They would have to get the feeling for the electronic parts and work very carefully, as the fabric could be damaged while inserting the electronic device. They would have to be trained regarding cable routing and special handling of each piece of electronic. We would need additional staff for production of Profitex Jackets, which would be in charge of this only, to avoid mistakes.
Delivery
We have to think about special deliveries. A jacket with electronic device needs a special package, a cardboard with padding material inside. The receiver of the delivery has to get a functional jacket.
Stock-keeping
We would have a stock of Profitex Jackets, but electronic devices would be stored beside. An integration would be done right before delivering. Electronic parts have to be stocked in an adequate way.
Guarantee
It's also necessary to define the period of guarantee of the Profitex Jacket due to every single electronic part which is integrated.
Additional services
Cleaning service
An important issue to discuss is the washability of the jacket. Integrated electronic parts, which are not resistant against water, would have to be removed beforehand and be integrated again by trained staff. By entering through the zip you have to be able to reach the textile buttons which are located in both sleeves. Regarding this issue, we designed all bags in a way they can be opened easily.
Repair center
A repair centre, especially for the Profitex Jacket, would be necessary. People who can determine damages of the garment and of the electronic devices too, would be necessary. In a professional way, like within the cleaning process, the electronic parts have to be removed and integrated again. That would mean the responsible persons can handle all electronic devices - or send it back to the suppliers.
Fitting with Profitex jackets
After placing an order at Texport, we normally organise a fitting with the fire brigade. We would produce several sizes of the Profitex jacket and the firemen can try them on to find their perfect size.
Work Package 4 - Verification
Objectives
Within this work package the whole testing operations and activities of the project will be hosted.
Starting from defining the conditions, the single components, the lab examples and the system will be tested.
Description of Work
Task T 4.1 Definition of test standards
Within T 4.1 test standards for evaluating the whole project will be chosen and / or defined.
Results achieved in Task
With the development of this task is fulfilled the initial objectives where identified in a series of standard and nonstandard tests for the characterization of materials, components and systems that form part of Profitex Personal Protective Equipment.
Description of Work
Task T 4.2 Testing of single components
Each component manufactured in T2.2 will be tested regarding technical, safety, functional and physiological aspects. The standards defined in T 4.1 and listed in D 4.1 will be used.
Results achieved in Task
Task T 4.3 Testing of the lab examples
Based on the outcome of D 4.2 as well as D 1.1-D 1.4 single components were chosen to build lab examples. Within this task the lab examples will be tested and evaluated, also according the defined standards from D 4.1.
Results achieved in Task
Following table lists the elements that systems have been tested and their comments thereon. It lists the components that are integrated into firefighter's jacket, and peripherals that are part of the system.
Description of Work
Task T 4.4 Testing of the system
As a next step the system as a whole will be tested in regard to the overall functionality. The test standards from D4.1 will be used.
As result of T4.4 a report of all test results will be given (D4.5).
Results achieved in Task
According to D4.4 the current version of the system got a positive feedback by the involved fire fighters. Furthermore, the lab-based tests (D 4.2-4.3) show that the textiles and methods used for making the clothing are suitable for the application of ProFiTex.
Description of Work
Task T 4.5 Evaluation
All results of WP4 gained up to this point (D4.2 D4.3 D4.4) are evaluated against the user and system requirements (D1.3) as well as the test standards (D4.1). The overall performance of the system will be reviewed this way to make sure that the system fulfils all requirements and will not be an obstacle but a support for each firefighter using it. If necessary, last system adjustments might be carried out.
Results achieved in Task
The results of this task were reported together with the results of Task 4.4. That was regarding the scheduled testing session within the mid of the project. The final testing is scheduled at end of September, 2012. Accordingly, the evaluation results will be reported.
Work Package 5 Certification
Objectives
Testing, evaluation and preparation for certification of the system
Description of Work
Task T 5.1 Testing and evaluation under realistic conditions
The complete system will be tested in an exercise mission under controlled, but realistic conditions in the training center. Professional firefighters as well as volunteers will use the equipment. All relevant data will be recorded and the performance monitored.
Results achieved in Task
D4.7 and D5.2 have joined in the D5.3 with given the similar characteristics of these two tasks have decided to incorporate collect activities into a single deliverable.
The main results obtained under this task are:
A map of "Certifiable" parts of Profitex System, and the main features of this certification. Have been identified the elements that are subject to certification and what not. For this three components family has established: 1. Garment integrated components, 2. Peripheral components created in the project, 3. Commercial peripherals used in Profitex System.
- A certificate of the system parts that are in final version (or final working prototype) state.
- Certification recommendations for Profitex system parts that are in "non-final version". Will feature a series of protocols for improvement functional components (or devices) that are not in a final version. Will consider recommendations for future certification.
Profitex project proposal the following way in four steps:
- Identify modules of technology integrated on garments and its components
- Identify which components have a individual CE as market product.
- Certification of components without CE
- Final Module certification and CE marking.
Potential Impact:
Scientific articles and Press releases
General introduction articles to the project carried out for publication in mass media, presenting the scientific objectives of the project to raise interest in the scientific community, and periodic publication of the progress of the project
Brochures
In order to complement the dissemination activities for the purpose of promoting the exploitation of research results with activities aimed at a non-specialised general audience and to provide information on the nature of the activities carried out during the project and the benefits to the industrial sector, the partners elaborated general information such as flyers or posters presented in conferences and exhibitions.
Website
The project web site including informations about the project, objectives and expected out-comes, is been regularly updated. The Profitex website was developed and furnished with essential informations (e.g. project description) from the very beginning. The Profitex Home-page provides continuous project information such as: Introductory information, activities, partnerships, events
Geographical Coverage
The geographical focus of the dissemination activities within the time frame of Profitex is been carried out both at national and international level:
- At a national level partners, used their existing industry contacts to disseminate Profitex and promote the take-up of the developed technology.
- At International level, Profitex is been promoted in conferences providing contacts with relevant sectorial organizations.
The IPR Tool has been organized into two main sections:
- Learning section
- Innovation Management Section
Project website: http://www.project-profitex.eu