Final Report Summary - MYCOPTER (Enabling Technologies for Personal Air Transport Systems)
Executive Summary:
The existing problem of traffic congestion combined with its anticipated growth presents a major challenge for future societal mobility and economic growth. A promising solution that combines the best of air- and ground-based transportation to overcome these problems is to establish a personal aerial transportation system (PATS) based on Personal Aerial Vehicles (PAV).
The myCopter project is a unique integration of social investigation and technological advancement to establish candidate enabling technologies for a PATS. Experts from across Europe have developed advanced technologies for automation, handling qualities and human-machine interfaces for PAVs, and have performed socio-technological evaluations to assess the impact of a PATS on society. The outcomes of this project provide a stepping stone for future endeavours aimed at moving personal transportation into the air.
The myCopter project has demonstrated that a personal aerial transportation system can become a reality, given appropriate technological advancements and socio-technological considerations. Innovations in vehicle automation, control augmentation and display interfaces have enabled pilots with extremely limited flight experience to safely operate a simulated personal aerial vehicle (PAV). Nonetheless, continued efforts are required to make a PATS that can be used by the general public a reality.
The outcomes of the myCopter project identify the next steps towards a PATS. The focus should be on development of real-world implementations of the automation and augmentation technologies required to bridge the skills gap between a highly-trained pilot and the average car driver. Just as importantly, several key socio-technological issues need to be addressed, including the legal and certification issues surrounding PAV automation and operation.
Project Context and Objectives:
The existing problem of traffic congestion combined with its anticipated growth presents a major challenge for future societal mobility and economic growth. A promising solution that combines the best of air- and ground-based transportation to overcome these problems is to establish a personal aerial transportation system (PATS) based on Personal Aerial Vehicles (PAV).
Commuter traffic causes peak loads on ground-based traffic systems, with traffic jams as a result. To alleviate this situation, an idea is to use the airspace for personal transportation. The myCopter project combines the old dream of a flying car with the need for changes in our current transportation system. Besides looking at desirable technological developments for such a PATS, the project partners are especially interested in the associated challenges that are connected with the realisation of such a system from a technology assessment’s point of view.
The aim of myCopter is to determine the social and technical aspects needed to set up a transportation system based on PAVs in today’s society. The project focuses on three research areas: 1) the human-machine interface and training, 2) automation technologies, and 3) the socio-technological environment that surrounds PAVs. MyCopter investigates the idea of a PATS based on small personal aerial vehicles flying short distances between home and the workplace that take off and land vertically and that can be used in densely populated areas.
The interaction between the pilot and a PAV is of crucial importance and depends on the dynamic characteristics of the PAV itself. Therefore, a generic flight dynamics model has been studied that simulates the dynamic behaviour of a PAV. Furthermore, PAVs need an intuitive human-machine interface (HMI). Within myCopter, we investigated the use of a haptic shared control in combination with a highway-in-the-sky display. Such an HMI shows the current flight trajectory in a perspective display and provides pilots with haptic guidance forces on a sidestick based on this trajectory.
Navigation in cluttered environments is crucial for safe flight. PAVs need to be able to find their way, navigate to a destination, avoid obstacles and mid-air collisions with other traffic, and find suitable landing places. These concepts have been studied within the myCopter project by taking novel approaches such as navigation based on vision information from cameras, collision avoidance based on simulations of human crowds and landing place selection based on monocular vision information.
The expectations of future users of PAVs cannot be ignored, as flying vehicles will have a large impact on society – or none at all, if they are not accepted. In the myCopter project, we investigated the technological issues surrounding PAVs, especially regarding the identification of technical and operational challenges. Furthermore, we have performed qualitative research into the expectations of potential users towards a PATS and how society might deal with this new aerial traffic system.
The technologies developed in the myCopter project have been demonstrated on various test facilities. Advances in automation technologies were implemented and tested in unmanned aerial vehicles and prototypes for HMI and training scenarios were investigated on fixed-base simulators. We have also implemented some aspects of these technologies into the Flying Helicopter Simulator, operated by DLR, the German Aerospace Center.
The myCopter project is a unique integration of social investigation and technological advancement to establish candidate enabling technologies for a PATS. Experts from across Europe have developed advanced technologies for automation, handling qualities and human-machine interfaces for PAVs, and have performed socio-technological evaluations to assess the impact of a PATS on society. The outcomes of this project provide a stepping stone for future endeavours aimed at moving personal transportation into the air.
Project Results:
**WP2: Flight simulation/training**
This work package (WP) developed a methodology to assess the handling qualities requirements for Vertical Take-Off and Landing-capable Personal Aerial Vehicles (PAVs). It is anticipated that such a PAV would be flown by a ‘flight-naïve’ pilot who has received less training than is typically received by today’s general aviation private pilots. It was found that the methodology used to determine handling requirements for a PAV could not therefore be based entirely on existing rotary-wing best practice, i.e. the use of highly experienced test pilots in a conventional handling assessment limits the degree to which results apply to the flight-naïve pilot. This WP developed alternative methods to assess the handling qualities requirements of a PAV based on both the subjective and objective analysis of performance and workload of flight-naïve pilots in typical PAV tasks. A highly reconfigurable generic flight dynamics simulation model that was used to validate the methodology was also developed. Results that highlight the efficacy of the various methods have been achieved and their suitability for use with flight naïve pilots demonstrated. The methods developed off the possibility of providing a means to quantify the current subjective methods used for professional pilots.
The WP undertook research to develop handling qualities guidelines and criteria for a PAV. The objective was to identify, for varying levels of flying skill, the response type requirements in order to ensure safe and precise flight. The work has shown that conventional rotorcraft response types such as rate command, attitude hold, and attitude command, attitude hold are unsuitable for the anticipated population of PAV pilots. However, so-called augmented response types such as translational rate command and acceleration command, speed hold permit ‘flight naïve’ pilots to perform demanding tasks within the required precision repeatedly. This is an interesting result as the current standards indicate that a rate command system should be sufficient for professional pilots to achieve the highest level of handling qualities in a good cueing environment but, as the environment degrades to its worst level, a translational rate command is required. PAV pilots require this highest level of vehicle augmentation for the good cueing environment but require no further augmentation as, for instance, the visual scene is degraded.
This WP also undertook research activities to development the training requirements for pilots of PAVs. The work included a Training Needs Analysis (TNA) to determine the skills that would need to be developed by a PAV pilot and the development of a training programme that covers the development of the skills identified by the TNA. The effectiveness of the training programme has been evaluated using the first three Levels of Kirkpatrick’s method. The evaluation showed that the developed training programme was effective, in terms of engaging the trainees with the subject, and in terms of developing the skills required to fly a series of PAV-mission related tasks in a flight simulator. The number of hours training required in a simulator indicates that they would not be prohibitively large when compared to, for example, learning to drive a car.
Motivated by simulator test subjects who expressed discomfort during certain landing profiles that were tested as part of the project, the WP also studied the design and assessment of a "natural feeling" landing profile to guide PAV occupants from cruising flight, down an approach path, to bring the vehicle to a successful hover, in other words, to ‘park’ the vehicle. The development of the new profile was motivated from the point of view that ‘natural-feeling’ cues are related to the physiological cues presented during a visual landing. As such, test subjects with little or no prior flight experience flew simulated approaches to a hover, following limited instruction in the use of a vehicle model. It was found that, despite the wide range of skills and experiences that the participants brought with them to the testing, the approaches that they flew were all broadly similar and could be grouped into three distinct phases. In fact, the approaches heavily resembled those approaches flown by skilled pilots in an earlier NASA study. This previous work in this field, as well as tau theory (a theory from psychophysics that posits that animals in the natural world guide themselves around the cluttered environment that is the Earth’s surface using the direct perception of time-to-contact to an object or surface) was used to design an idealized approach profile based upon the simulation results. It was found that the PAV occupant’s preference for the approach profile depended upon whether or not they were flying the vehicle themselves. The results indicate that the PAV occupants preferred a descent with a constant deceleration profile for an automated landing but for a manual landing, i.e. one where they were flying the vehicle themselves, the designed ‘natural’ profile was preferred.
Finally, this WP also investigated the use of novel control systems designed to provide a safe and reliable method for the control of a future PAV. As it is envisaged that most PAV ‘pilots’ will be familiar with driving a car, the use of response types and controls derived from road vehicles was investigated. Objective and subjective techniques were used to quantify and qualify the applicability of automobile-like response characteristics using traditional helicopter control inceptors modified to behave somewhat like automobile controls, e.g. using foot pedals to control an Acceleration Command, Speed Hold system. Additionally, the effects of eliminating vehicle pitch and roll dynamics were investigated to determine whether this results in a reduction of workload for non-professional pilots. The results suggest that, particularly for the most inexperienced of pilots, the automobile-like configuration is more suitable to allow them to control a PAV than an augmented set of helicopter-style response types. This was shown through increased performance, and a reduction in subjective workload ratings. Improved Handling Qualities Ratings (HQRs) were also obtained in the automobile-like system for the tasks undertaken. Overall, removal of the pitch and roll dynamics was not found to significantly affect task performance in the automobile-like system, but their absence resulted in a decrease in performance for the rotorcraft-style response type configurations tested.
**WP3: Human-Machine Interface for controlling a PAV**
Controlled experimental environments provide the perfect means for developing and testing novel concepts for human-machine interfaces. Within myCopter, we have developed experimental facilities to study pilot behaviour with novel human-machine interfaces in an active control loop. In these experimental facilities we have performed basic research into novel approaches for haptic guidance and human factors (for example, cognitive workload) for personal aerial vehicles. Our studies cover a range of topics such as the design of haptic feedback systems, workload assessment with physiological metrics, looming warning signals and the development and identification of dynamics models of helicopters to aid the implementation of real-world augmentation strategies for non-expert pilots.
*Haptics research*
Our research has led to the development of a prototype human-machine interface (HMI) for personal aerial vehicles that is based on the combination of a haptic shared control framework and a Highway-in-the-Sky display. We investigated whether this combination could result in improved performance for non-expert pilots flying a personal aerial vehicle. Various representations of a flight trajectory in a highway-in-the-sky display were evaluated. It was found that a tunnel and a wall representation led to the best performance, whereas a highway representation resulted in worse performance and higher control activity and effort. Haptic guidance cues on the sidestick allowed pilots to achieve better performance with lower control activity.
The prototype HMI was demonstrated to the general public and the press. Anybody with at least some driving experience could easily get accustomed to the PAV flight dynamics and use the supplied visual display and haptic guidance cues to follow a flight trajectory without large deviations. Participants reported that they found the guidance forces in the various degrees of freedom very intuitive.
Our prototype HMI was based on more fundamental research focusing on the influence of haptic feedback on pilot control behaviour. Two major approaches were identified, the indirect and the direct approach. The indirect approach provides haptic feedback cues that a pilots needs to oppose, while the direct approach provides guidance cues that a pilot needs to follow. Our research has shown that the indirect approach outperforms the direct haptic approach during aircraft pitch angle tracking. This higher performance is due to pilots changing the neuromuscular settings of their arm (admittance) from compliant to stiff when they were provided with indirect force cues as compared to direct guidance cues. However, a disadvantage of the indirect approach is that pilots need to invest more energy into the control task as they have to adopt a “stiff” control strategy by co-contracting the muscles in their arm.
Our novel HMI combines automated control with manual control. The haptic feedback system assists vehicular steering by sharing manual control with the human operator and guiding the operator towards an optimized trajectory. Furthermore, such an approach allows him/her to augment, comply with, or resist the haptic guidance forces according to his/her preferences. Therefore, haptic feedback forces can be used to improve performance and increase safety. Nonetheless, the human operator may not always benefit from the haptic support system. Depending on the amount of haptic feedback, the operator might oppose this haptic assistance or demonstrate an over-reliance. Therefore, we have investigated the levels of haptic assistance that would optimize shared control between the human operator and the vehicle and have shown that pilots can optimise their response in response to the provided haptic support.
*Human Factors research*
Our research has also focused on the human factors associated with PAV flight (e.g. cognitive workload) by employing non-obtrusive real-time methods such as gaze tracking, electro-dermal response, heart-base measures and electro-encephalography (EEG) to estimate the perceived difficulty of PAV flight tasks.
One example of this work is that by tracking the eye movements of pilots in a cockpit environment, we were able to evaluate how humans coordinate their need for visual feedback during flight control. This has allowed us to identify the expected response latencies of the pilot to the appearance of peripheral targets. In addition, we have identified a behavioural feature that distinguishes novice pilots for their control performance: eye-movement efficiency across visual instruments. We have also verified the relative importance of different cockpit instruments in order to complete a PAV scenario. With this in mind, perspective flight-path displays should be designed to emphasize these informational aspects, according to their gaze frequency of on a traditional flight deck.
Another example of our research deals with delays between user input and the system’s reaction in control tasks, which have been shown to have a detrimental effect on performance. This is often accompanied by increases in self-reported workload. We have identified physiological measures that correlate with pilot workload in a PAV that suffered from varying time delays between control input and vehicle response. For this purpose, we measured the skin conductance and heart rate variability during flight manoeuvres in a fixed-base simulator. Control error and the self-reported workload increased with increasing time delay. Skin conductance and input behaviour also reflected corresponding changes. Therefore, these physiological measures are sufficiently robust for evaluating the adverse influence of system delays.
Commonly, the extent of mental workload can be estimated by self-reports (e.g. with the NASA Task Load Index (TLX)). Within myCopter, we have been working towards evaluating workload physiologically in terms of how a primary task taxes a common and limited pool of mental resources, to the extent that it reduces the electroencephalographic (EEG) responses of the brain to a secondary task while performing a primary flying task. Our experiments show that changes in the EEG response of a pilot can be used to identify the control effort required in a specific flight task, but that it is not sufficiently sensitive to provide a graded response across different levels of disturbances. We are still performing further research to improve upon the sensitivity of the EEG metrics.
Finally, we have also investigated the effectiveness of warning cues. Automated collision avoidance systems promise to reduce accidents and relieve the driver from the demands of constant vigilance. Such systems direct the operator’s attention to potentially critical regions of the environment without compromising steering performance. Auditory warning cues with rising intensities are claimed to be especially salient. By evoking the percept of an approaching object, they engage a neural network that supports auditory space perception and attention. Our experiments have shown that we are indeed aroused by and faster to respond to ‘looming’ auditory signals, which also increase heart rate and skin conductance activity.
*Augmentation strategies*
Finally, we have also developed dynamic models of helicopters to aid the implementation of real-world augmentation strategies to aid non-expert pilots. We are investigating the interaction between a pilot with limited flying skills and augmented vehicles that are as easy to fly as cars are to drive (for drivers with limited experience). We focus on light helicopters as these best reflect the properties of a vehicle that could be used in a personal air transport system. Therefore, we are identifying a rigid body dynamic model of a light helicopter in flight tests, such that we can implement augmented system features that allow a pilot with limited flying skills to reach similar performance as a highly-trained pilot. This will be tested in handling qualities and human performance evaluations in piloted closed loop control tasks, with and without the augmented systems.
Similarly, we have developed a non-linear helicopter model for simulating realistic flight scenarios. Commonly, non-linear models are developed for unmanned small-size helicopters, but nonlinear models for full-size helicopters are not very common, due to the difficulty of accurately implementing the components of the vehicle and obtaining reliable aerodynamic parameters. We implemented and validated a model by performing test manoeuvres with the help of closed-loop controllers and a helicopter pilot that flew the model in a simulator. This non-linear helicopter model will allow us to investigate novel non-linear augmentation strategies for supporting inexperienced pilots in PAVs.
**WP4: Control and navigation of a single PAV**
*Vision-aided navigation*
This work package addressed the automation challenge for a single PAV with the aim of developing the intelligence towards autonomous navigation of a Personal Aerial Vehicle (PAV) as defined in the context of the myCopter project. In general, a PAV flight can be divided into three phases: 1) take-off, 2) cruise flight and 3) landing. In order for automation to work in all phases, a good estimate of both the state of the vehicle (position, velocity, orientation) and the environment are essential. During cruise flight, localization of the vehicle can usually be done by GPS, while for take-off and landing, novel camera-based approaches were proposed and developed, as GPS is not a reliable service in close proximity to urban structure.
We investigated flight-path planning and a navigation display for use during cruise, which resulted in a so-called “Highway In The Sky” (HITS) Display: Conventional helicopter cockpits feature an instrument panel with a variety of instruments that the pilot has to check continuously. The pilot has to connect the individual pieces of information in his mind in order to get a sense of the behaviour of his vehicle. Training and experience is required for effectively piloting a helicopter with the information from a conventional instrument panel only. For PAV pilots who will have only a minimal training compared to today’s helicopter pilots, a more intuitively understood instrument panel must be designed. A strong candidate for this purpose is the highway-in-the-sky (HITS) display. The HITS display is an easy to use navigation display. A tunnel depicts the flight trajectory. The tunnel shape can be selected to represent walls, a u-shape, a box or a highway. Overlaid to the three-dimensional design is a two-dimensional primary flight display (PFD). This features conventional indications for attitude, heading, altitude, airspeed and torque. Additionally, optional target indicators, so called bugs, are implemented for flight parameters such as airspeed and altitude as well as pitch angle and torque. These bugs have a rectangular shape with cut-out triangle in order to provide the pilot with information on the acceptable deviation from the target value.
These phases for take-off and landing are considered to take place in urban environments, where reliable GPS service cannot be guaranteed anymore. We have used cameras in combination with an Inertial Measurement Unit (IMU) as the main sensor setup for localization in the control loop. We identified perception and state estimation as the key components that enable autonomous or automated flights in GPS-restricted areas. Only if these components provide a good estimate of the current state of the vehicle, controllers stabilizing the vehicle can work reliably and fly to waypoints or tack trajectories created by a path-planner. Using camera(s) as the only exteroceptive sensor(s), we fuse inertial measurements to achieve a self-calibrating power-on-and-go system, able to perform autonomous flights in previously unknown, outdoor spaces. Our framework achieves Simultaneous Localization And Mapping (SLAM) with previously unseen robustness in onboard aerial navigation for small platforms with natural restrictions on weight and computational power. Successful navigation and control of this real system has been thoroughly tested in real flight experiments, and under a variety of challenging conditions. All flights were performed autonomously using visual cues directly in the control loop, with only high-level waypoints given as directions. The state estimation framework was furthermore improved to be able to process delayed, relative and absolute measurements from a theoretically unlimited number of different sensors and sensor types, while allowing self-calibration of the sensor-suite online. This work done in myCopter was amongst the first that showed successful navigation of a Micro Aerial Vehicle (MAV), solely based on visual and inertial measurements and their fusion.
As visual-inertial based localization and control require certain excitation in order to render all their states observable, we went one step further and developed a path planning method, taking these requirements into account. The method also incorporates the ability of the cameras to localize reliably and plans safe paths to the selected goal. That is, we seek to find not only a short path to the destination, but an informative path where the states monitored in our system are best observable at all times, and where the goal can be reached within some given confidence area. As an example, it is safer to fly a small detour, in case a direct path to a landing place does not exhibit sufficient visual features.
In order to be able to follow the paths generated by our planning method, we have developed a simple but powerful control approach for the MAV, which can be easily deployed on computationally restricted embedded hardware, and increases bandwidth. This controller is kept generic in order to be applicable to a variety of rotorcraft.
In summary, we have developed novel methods for control and navigation of aerial vehicles. These methods were demonstrated successfully on both the simulation facilities as well as Micro Aerial Vehicles. MAVs are great research platforms for performing experiments, and are believed to have properties similar to a future envisioned PAV. Even if not immediately deployed on a PAV, many of the developed methods are already applicable to current systems.
*Landing place selection and assessment *
When it comes to the transition from cruise to landing, we have investigated methods for automated visual landing place selection and assessment in man-made environments. We have developed an approach for automated landing place assessment in man-made environments. With our approach candidate landing sites are detected as featureless, regularly shaped areas in the image that typically characterize man-made landing structures and we devise an algorithm to compute them efficiently. Landing sites are then selected by searching for cycles in a line fragment graph formed from lines detected about the contours of candidate image regions, using learned measures of object appearance and geometry.
Our approach for candidate landing site detection is based on finding Maximally Stable Extremal Regions (MSERs) that typically correspond to dominant, featureless regions in the image. They are computed by searching for stable regions across varying image thresholds, which for reasonably sized images can be done very efficiently. We further apply Hough line voting to measure the shape of each MSER and detect MSERs exhibiting a polygonal shape. We have demonstrated this approach for the detection of both prepared and unprepared landing sites including airport runways, building rooftops and rural fields, where it was shown to obtain near real-time performance in the case of single frame low-resolution images. A paper detailing this work has been published in the Machine Vision and Applications journal. We have also developed a multi-feature version of our algorithm and demonstrated it for landing site detection on sequences captured from the FHS helicopter.
Our algorithm for landing site selection is based on finding cycles in a line fragment graph. More specifically, given the detected MSER regions and line fragments we construct a weighted graph, and search for contour regions as cycles in this graph using a learned cycle scoring function that can be used to select landing sites of a particular type. Importantly, unlike previous graph cycle object detection methods, our approach can be used with a wider variety of cycle scoring functions and image features to result in an increased accuracy. A paper detailing this work has been published in the European Conference on Computer Vision.
Our work on vision-based geographic localization has focused on algorithms for fast and accurate image matching and indexing in large environments. Toward this end, we have developed a boosted binary keypoint descriptor that is not only accurate to match but also highly efficient to compare and store.
In our initial work we proposed a boosted floating-point descriptor (FPBoost), also referred to as Low-Dimensional Boosted Gradient Maps (L-BGM), that applies boosting to learn a visual similarity measure from labelled pairs of similar and dis-similar image patches and optimize over the pooling configuration of the resulting image descriptor. FPBoost was shown to rival the performance of the state-of-the-art floating-point descriptors and provide a significant improvement over SIFT. We then extended this idea to develop a boosted binary keypoint descriptor (BinBoost) that greedily optimizes over a set of binary hashing functions to result in a descriptor that is not only accurate to match but also very fast to compare and efficient to store. BinBoost provides a similar accuracy to the state-of-the-art floating-point descriptors at a fraction of their matching cost, and results in an increased accuracy over the state-of-the-art binary approaches.
**WP5: Navigation in the air and the interaction with other traffic**
*Collision avoidance *
In this work package, we tackled the problem of mid-air collision avoidance in dense aerial environments occurring in the myCopter scenario. High densities of agents decrease the time to react before a collision and thus need a quick response from the agent, limiting negotiation time resulting from collaborative strategies. Furthermore, finite communication bandwidth limits information exchange between agents, leading to an incomplete view of the environment. A reliable collision avoidance strategy for high densities traffic should therefore be reactive, fast and robust to unforeseen circumstances. Centralized strategies suffer from a single point of failure and bottlenecks. The need of constant recalculation and time to transmit solutions prevents the use of such methods in these dynamic and dense environments.
With the use of local sensing and finite range communication devices, the knowledge of the environment is incomplete and global path planning methods would therefore potentially fail by not considering a future threat that they are not aware of at the time of planning. Reactive approaches where the controller adapts to the actual situation are more suited to address this problem.
With passengers on board, the question of comfort during the flight has to be addressed. The commuting journey should be as smooth as possible and not exceed physiological comfort limits.
To alleviate these challenges, we proposed a crowd-inspired, reactive approach to address collision avoidance in dense aerial traffic, and developed an extension to improve passenger comfort. Extensive simulations have been carried out with more than 200 simulated PAVs to evaluate travel time and relative comfort. Additionally, we show that the proposed strategy can be implemented on unmanned flying robots in real-time with minimal computing requirements in an experiment with ten robots flying outdoors. We developed the entire platform used to perform these tests. To facilitate safe experimentation with many robots, the Micro Aerial Vehicle was designed to be very lightweight (<500 grams), very easy to assemble and repair, and low cost. We have now a fully working quadrotor and autopilot, which is easy to build based on 3D-printed parts and low-cost off-the-shelf components, with full access to the software. The software and the hardware components will be open-sourced.
In addition, we showed how this framework could be extended to deal with non-flying zones. Non-flying zones are geographically defined zones were no flying vehicles should overfly. Cities’ downtown, large airports, nuclear plants and military basis are examples of places that no one is allowed to fly over.
Finally, we demonstrated how the standard Reynolds flocking rules could be modified in order to deal with agents with individual goals. Evolution came up with a simple solution to navigate safely in dense environments. In Nature, flocks of birds, school of fishes, swarms of insects, human crowds and bacteria are examples of collective behaviours emerging from local control and local sensing. It can be observed that large-scale behaviours coming from self-organization process emerge spontaneously from only local interaction and local sensing. Reynolds proposed some simple rules to describe the flocking of birds. However, these rules apply to agents having a common goal such as migration. Therefore, with standard Reynolds flocking, the agents would have to alter their trajectory too much in order to stay with the flock. On the other hand, with our solution, flocks of PAVs will be naturally joined and left depending on the flight direction of each PAV. With this framework, the advantages of spatial closeness and the egoistic behaviour of PAVs will be optimally combined to guarantee a safe and direct navigation to each one’s goal.
*Vision-based detection of other PAVs*
We developed approaches for the detection of both collision course and general flight of neighbouring PAVs. To detect PAVs on a collision course, we not only use their distinctive appearance but also their unique motion pattern. Toward this end, we employ a spatio-temporal Histogram-of-Oriented Gradients (HOG) feature along with a linear Support Vector Machine (SVM), evaluated using a sliding window approach over multiple spatial window sizes. PAVs on a collision path are then identified as those that appear stationary and growing in size. We evaluated our approach using a video dataset. Exploiting the temporal pattern of aircraft on a collision course results in a highly accurate performance that significantly outperforms 2D detection with or without temporal smoothing. When additional computational resources are available an even further increase in accuracy can be achieved using larger window sizes.
Under general flight both the motion of the aircraft and the neighbouring aircrafts must be compensated. We first compensate for camera motion using local feature matching. Neighbouring aircraft are then detected using a sliding-window linear SVM classifier with spatio-temporal HOG features, where the frames of each spatio-temporal window are normalized according to the motion of the neighbouring aircraft, whose motion is estimated using a learned mapping function. Our approach achieves a significant improvement over 2D detection alone with a 2D HOG classifier or boosted trees, the comparison with not aligned 3D HOG highlighting the importance of aircraft motion correction.
Both of our approaches for the detection of neighbouring aircrafts require annotated aircraft images that can be costly to collect. We therefore developed a synthetic data generation approach to help alleviate manual annotation effort. Provided a 3D model of an aircraft, for each annotated aircraft image we generate synthetic images by rendering the model and matching the rendered aircraft image to the real one by optimizing over its rendering parameters. Our approach results in a significant improvement over labelled-only performance, the most significant improvement resulting when using the supervised image matching criteria of the corresponding classification scheme.
**WP6: PAV operational system concepts**
The objective of this work package (WP) was to develop the concept of a vehicle and the supporting systems that will fit the needs of the overall call for a PAV.
The first achievement of this WP was the development and analysis of design features and options for a PAV and the supporting system or infrastructure. This started with descriptions of the intended use case scenarios for PAVs and the intended users. PAVs are foreseen to be commuter vehicles in the first place. They shall provide point-to-point connections for typical commuting distances and shall be competitive to existing ground-based transportation methods. PAVs are supposed to be available to the general public, which leads to new requirements regarding the qualification of pilots. Obtaining a PAV license must become clearly less complex in terms of necessary knowledge, training time and financial investment than keeping a helicopter pilot license today (e.g. PPL-H).
The consortium agreed upon initial design features as a common basis for further development of a reference PAV. A preliminary design study based on helicopter theory methods and experience from pre-design of conventional helicopters was conducted (e.g. power consumption calculations were used for estimating necessary battery capacity and weight). Two development phases have been elaborated regarding the necessary level of automation, which cover different automation functionalities. During the first phase PAVs would be equipped with certain assistance functions, e.g. for stabilization and altitude, speed control or GPS navigation, but the PAV user would still need to fly manually in certain flight phases. The second phase would then assume to have fully automated PAVs. The user would not be pilot anymore but could be understood as human payload in a fully autonomous air vehicle.
This WP also undertook research in the field of a novel easy-to-handle human-machine interface suitable for future PAV pilots. The control concept of automobiles is well known to the general public and the related inceptors have barely changed over the past century. Although new controllers like joysticks are technically feasible, all of the modern production vehicles rely on the conventional arrangement of steering wheel, accelerator and brake pedals, gear stick, and optionally clutch pedal for manual transmission. A driver’s license holder can intuitively connect the usage of these controls to the movement of his car. On the other hand, conventional helicopter controls are not at all intuitive for non-expert pilots. There exist PAV prototypes that feature different steering concepts for different phases of operation, such as a steering wheel for ground operation and conventional gyrocopter inceptors for flight. This means that the user must learn to use two different control concepts and even switch the inceptors for transition from road to air mode. The approach within this project was different and our goal was to simplify usage instead of multiplying the amount of inceptors. The average car driver has gained extensive experience in steering wheel control of ground vehicles. We wanted to make use of this prior knowledge and extensive amount of training to transfer the benefits to rotorcraft control. Therefore, the intuitively understood steering wheel was used as central element of the new rotorcraft control concept. Together with the advanced response types for PAVs developed in WP2, a car-like steering concept for rotorcraft was developed and tested.
These PAV technologies were integrated and tested in a motion-base simulator. This allowed us to perform handling qualities evaluations under conditions close to real flight. Also certification activities were undertaken such that the novel steering wheel concept for PAVs could be tested in-flight. Unfortunately, a technical failure occurred on the basis aircraft that grounded the research helicopter until the end of the myCopter project, which prevented us from pursuing a full flight test regime, but flight tests are still planned once the research helicopter is back in operation.
The baseline configuration for the investigations in the simulator flight trials was the hybrid flight dynamics PAV model with different characteristics dependent on the flight condition, as developed in work package 2. Five pilots participated in a handling qualities study where the four mission task elements had to be completed (Hover, Hovering Turn, Vertical Manoeuvre and Slalom). The baseline configuration with conventional pilot controls received average handling qualities ratings (HQR) in the desirable Level 1 regime. The baseline configuration was then compared to a configuration with activated model-based control (MBC). This control technique was needed for in-flight simulation on the research helicopter. A quantitative analysis showed the model following capabilities to be acceptable to good. Deficiencies were found in the hover where drifts occurred. Evaluations with five pilots showed that the MBC configuration was flyable with handling qualities ratings in the Level 2 regime (adequate).
Finally, a steering wheel inceptor has been implemented as the primary inceptor for PAV control. The steering wheel replaces the conventional centre helicopter stick, which is a novelty in rotorcraft control. The control axes are re-defined to resemble a car-like control concept. Thus, the steering wheel is used to command a combination of yaw and roll inputs depending on the current airspeed. The pedals are used for controlling acceleration and deceleration along the longitudinal axis. The collective lever is used for height control. An additional 8-way switch in the centre of the wheel features precision manoeuvring functions. In the piloted evaluations the awarded handling qualities ratings for all mission task elements were better for the steering wheel configuration compared to the configuration with conventional inceptors. The workload, which was measured by the NASA Task Load Index (TLX), was also lower when the steering wheel was used. The only exception is the hovering turn. Here, the workload slightly increased due to the new sequential control strategy the pilots had to adopt. Although the pilots involved in the study rated the steering wheel configuration with a lower overall score on the System Usability Scale, the better HQR and TLX ratings prove the feasibility of a steering wheel control concept for rotorcraft, which was envisioned to be especially suitable for the pilots of future rotorcraft with highly augmented control laws such as PAVs.
**WP7: Exploring the socio-technological environment**
The main goal of this work package was dedicated to gaining deeper insights into the socio-technological context and the infrastructural environment of a future personal air transportation system (PATS). The operation of personal air vehicles (PAVs) raises plenty of questions about their potential impacts on society, and it is not clear what the expectations of society regarding PAVs actually are and how they will be met. Since PATS are in a very early stage of development, the demand for this form of transportation is still vague, as are people’s preferences for the design of PAVs and their associated infrastructure. It is also imaginable that a broad implementation of PATS may, at least potentially, have significant broader implications for a complex socio-technical system: mobility.
In order to explore these issues, at first a scoping phase was conducted with the aim to map the socio-economic environment of this new transport form, to identify the challenges and issues surrounding an actual realization of a PATS and its integration into the existing transportation system in Europe (see Deliverable D7.1 of the myCopter project). The potential opportunities, obstacles and challenges of PAVs were identified by an extensive literature research on topics such as technologies, regulation, safety, ATM, UAVs etc., supplemented by expert interviews. The key issues found to be relevant for a real world implementation of PATS into our current cities were safety concerns, including the weather issue, legal aspects (certification, insurance, airspace regulation etc.), technical and operational challenges such as automation and autonomy issues, Air Traffic Management (ATM), parking, storing and maintenance infrastructure, and socio-economic and ecological challenges such as energy and noise, acceptance and the aspects of system integration.
One of the key ideas of myCopter is to explore the potential of PATS/PAV to open the third dimension for commuter traffic and thus to reduce the negative impact of congestion in European metro areas. A general assessment of the overall impact of PATS on European road traffic appeared to be difficult for methodical reasons. The regional impact of PATS may differ considerably, depending on the respective local situation, since all cities/metro regions differ with regards to a number of factors such as spatial structure, structure of the local economy, socio-demographics, and many more. In general, one could imagine conducting site-specific studies for all relevant metro areas within the EU. But this would be far too complex for a single research project – and most likely also too early, given all the other uncertainties mentioned above. Therefore, it was decided to follow a more generic approach: Short narrative travel scenarios, describing stylized trips to and from work (addressing different settlement densities and architectures as well as autonomy levels/user qualifications) were developed and used to find categories of requirements which should be discussed and framed in order to clarify the vision and to develop a coherent “reference case” for the vehicle.
Coherence, here, is based on the understanding that the design parameters in the reference case cannot be set independently. A decision for one performance requirement, for example the seating capacity of the PAV, has an influence on other requirements of the PAV (internal dependency) like take-off weight. Some requirements are also strongly connected to the mission that the PAV shall provide for.For instance, cruising speed for PAVs has to be significantly higher than for cars in order to offset time losses during pre- and post-trip procedures and to remain competitive with regards to door-to-door travel times. With regards to vehicle size, it was decided to go for an option to transport either one person with some luggage located on the second seat or to provide transport for two persons – which reflects today’s realities in occupancy rates in commuting. This, in turn, permitted to reasonably assume a take-off weight of 450 kg. Calculations by the project partner DLR showed that PAVs of that size could be operated with a fully electric propulsion system, using either next stage batteries or fuel cells, and provide for a satisfying range.
As the PAV in commuting is competing with private cars and should be capable of being integrated into the existing transportation system, a car-like physical dimension was seen as wise in order to be able to use existing infrastructures for parking and more. As the Reference PAV is assumed to have VTOL abilities in order to be able to start and finish trips close to homes or in CBDs, respectively, only limited manoeuvrability - in order to enable access to storage facilities - but no “roadability” was foreseen.
Considerations on weather dependence started with the premise that, in the context of commuting, a PATS has to be available year-round – at least in principle. On the other hand, experience shows that this is virtually impossible to achieve, at least in most regions in Western and Central Europe. On order to illustrate the related challenges, a benchmark of 90 % usability year-round was deliberately set. This figure is well below the estimates for the usability of cars in the same region but still very ambitious for a PAV. Assessments based on General Aviation Forecast (GAFOR) data for Rhine-Main showed that this could only be reached if for some trips quite bad weather conditions with a visibility on the ground of less than 1.5 km and/or a cloud base height of less than 500 ft would be accepted by the user and could be safely provided by the system. These results illustrate that the topic of how to expand the operability of the PAVs into challenging weather conditions will have to be addressed further if PAVs should become a serious solution for everyday mobility. They also underscore that PAVs, in order to become a meaningfully usable part of everyday transportation, need to be technically able to provide more options than just those following visual flight rules (VFR).
In a similar manner, heuristics for a commuting scenario using a “stylized” European city were developed. It was shown that, assuming a number of approx. 300,000 people that commute every day into a major city, modal shares typical for European cities and a net substitution rate of 10 % of car traffic by PAV, an “automated” ATM for such a prototypical city would have to handle between 2,500 and 10,000 approaches per hour. Between 40 and 160 independent landing sites for PAV would be needed. Further assuming a conventional business model (“individual ownership”) and limited autonomy (no ability of fully automated flying) of PAV, this scenario indicates a required storage capacity for 7.000 to 20.000 PAV within the city. Especially the required capacity for storing temporarily unused PAV in a city centre would, in our opinion, substantially limit the diffusion potential of this technology. In other words, it would – among other reasons – support an implementation strategy that builds on fully autonomous “PAV on demand” which, like a driverless taxi, is able to pick up riders at home and drop them off at a number of dedicated sites within a city.
In order to do explore and understand the perspectives and expectations of potential users and/or citizens with regards to PATS development, three focus group interviews with laypeople were performed in the myCopter project and their outcomes were used to revisit the PATS concept as well as the specifications of the reference PAV (see Deliverable D7.3). The general technical feasibility of a PATS was not a major issue in all of the focus groups. Main issues identified in desktop research reappeared in the focus group discussions (like environmental impact, especially noise footprint and ‘fuels’, driver education, automation, availability, parking …) and were, in a way “hardened”. It was, for instance, strongly confirmed that the environmental footprint in general and in particular the energy consumption of the PAV must be as low and as renewable as possible. Therefore, the strong request for an electric propulsion concept for a PAV was further supported. Nonetheless, even after a lively debate about potential advantages, also doubts about the actual benefits remained among a number of the participants. A frequently discussed question was whether PATS would really improve the traffic situation on the ground or if every newly created mobility option and added capacity would not just lead to new traffic up to a point where everything – including the air space – would be overcrowded again. This mirrors a quite prominent but still controversial discussion in transportation planning, according to which many (urban) roads have latent travel demand and additional vehicle trips will occur if congestion is relieved – a rebound effect of infrastructure expansion also known as “induced travel”. Some perceived PAVs as “over-engineering”, stating that the level of automation needed to have PAVs would probably already solve congestions problem on the ground if implemented in cars.
But also new, additional aspects were mentioned that could not be investigated further in our study but might be subject to further considerations, like the usability of PAV “on the ground” (flying at 20 cm level), e.g. to integrate them in conventional road traffic under certain circumstances, since at least during a transformation period not all destinations could and would be equipped with dedicated infrastructures for PAV and hence a “dual use” of existing infrastructures both for PAV and cars should be considered. This would also address some concerns about the limitations to spontaneity and to integration into ‘real life’ trip chains that a PATS might incur. Beyond that, “more simple” options for early implementations were proposed. One idea was to use PAV like “rope-less aerial ropeways” – point-to-point connections in geographically challenging or already overcrowded areas where other options that rely on extensive fixed infrastructure are not technically and/or economically feasible. They would leave the technical system in the hands of professional operators (and reduce non-technical challenges like safeguarding proper maintenance challenges), would allow for testing PAVs in a controlled environment and solve pressing local problems. Comparable arguments may hold true for similar applications like island shuttles or valley hopping.
The greatest remaining challenge, nonetheless, is to decide about the automation concept of PAV and PATS. In myCopter, two basic concepts have been considered in parallel: fully autonomous and semi-autonomous (or colloquially: the elevator model and the driving pilot model). Fully autonomous PAVs were thought to perform all flying tasks without any interactions with the user. They would be able to move even without a person on board, allowing to disconnect pre- and post-trip processes from the user and to support new forms of mobility. The idea of semi-autonomous PAVs tries to link today’s routines of driving a car to a vehicle that is capable of traveling in three dimensions. In this case, assistive technology would be implemented on a vehicle that supports the driver in having longitudinal, lateral and vertical control while at the same assisting him in remaining in control, finding directions and avoiding collisions. Both concepts appear to be attractive to potential users, and even the focus group participants were, in a way, undecided about their likings. It was broad consensus that the preference for the degree of automation depends on the purpose of the trip, favouring full autonomy for daily routine trips (like commuting) and the opportunity to use a PAV more freely during leisure trips or “just for fun”.
The “automation question” was also a central topic of a subsequent expert workshop with academics, pilots and interest club members. Here, especially the challenges of designing semi-autonomous PAVs, their HMI and potential development paths towards full autonomy were discussed in greater depth. The experts argued that if the user would be foreseen (by the developers) to be responsible for taking over flight tasks in case of an unexpected situation, or a situation no longer manageable by the system, this would mean that the user would have a function like a professional safety pilot. They were very critical about a development philosophy that relies on the user as this kind of emergency backup. This role would be challenging even for well-trained pilots and obviously contradict the development goal of limited training requirements for a general PAV user. Instead, experts recommended considering a very communicative and transparent system which continuously informs the user about its current state as well as its intentions. For the semi-autonomous mode they suggested to permanently keep the user “in the loop” to avoid long delays before regaining full situation awareness.
This work package of the myCopter project has succeeded in providing first insights into the interdependence of the various factors in PAV/PATS development and realization but still was largely explorative. Future system studies will have to focus on a PATS concept with a further reduced option space. What can be said, nonetheless, is that the implemented level of autonomy of the vehicles will be a key design feature and decisive for future deployment strategies (and, in consequence, of more detailed assessment scenarios).
Potential Impact:
**Potential impact**
Several issues need to be resolved before a personal aerial transportation system can be realised in today’s society. The myCopter project aimed to make specific advancements in the user-centred design of human-machine interfaces and training for personal aviation, automation of personal aerial vehicles (PAVs), and the socio-technological assessment of the impact of a personal aerial transportation system on society.
Within the project we have evaluated several enabling technologies. Novel approaches to human-machine interfaces, such as haptic shared control and highway-in-the-sky displays, have been developed for use by non-expert pilots in highly augmented PAVs. We have shown that this combination can provide non-expert pilots with an easy-to-use control interface for flying a personal aerial vehicle.
We have also implemented a steering wheel in a helicopter that replaces the conventional centre stick. The control axes are re-defined to resemble a car-like control concept. Thus, the steering wheel is used to command a combination of yaw and roll inputs depending on the current airspeed. The pedals are used for controlling the longitudinal axis. The collective lever is used for height control and an additional 8-way switch in the centre of the wheel features precision manoeuvring functions. Experimental evaluations have shown that this concept a good alternative to conventional helicopter controls as pilot workload was decreased and handling qualities ratings increased in PAV flight tasks.
Furthermore, we have developed requirements for the Handling Qualities of future PAVS, together with a training syllabus for prospective PAV pilots. A training needs analysis for PAV flight resulted in the definition of five ‘lessons’, each focusing on a specific part of the PAV flight envelope and specific PAV manoeuvres. Several volunteer student PAV pilots have been trained for PAV flight with the training syllabus. They could complete the training in less than 5 hours on average, which highlights the efficacy of the developed training syllabus and PAV Handling Qualities.
Our approaches for navigation strategies for PAVs have shown that it is possible to reliably use vision-based information for navigating through cluttered environments. Also, it is possible to select landing places from monocular camera information with higher precision than with other methods. Collision avoidance can be performed in high-density environments for agents with individual goals by using strategies based on human crowd simulations.
We have identified user expectations towards a personal aerial transportation system and potential scenarios for PAVs. Through technology assessment methodologies, we offered an integrated view on the socio-technological environment that surrounds the personal aerial transportation system and highlighted pathways for bringing the implementation of a PATS forward.
The partners in the myCopter project believe that the outcomes of the projects will allow taking first steps towards the implementation of a personal aerial transportation system in today’s traffic system. Such a system could have a beneficial impact on our daily lives and could help solving issues related to current traffic and environmental problems. It would allow the possibility of direct travel between two points and therefore avoid people to sit in traffic queues, reducing travelling time and frustration. This could potentially save fuel and would reduce the stop and go traffic that wastes large amounts of energy. Travelling conveniently in the third dimension from door-to-door might allow for less congestion and would not require maintenance of support structures such as roads currently necessary for mass transit.
The myCopter project has demonstrated that a personal aerial transportation system can become a reality, given appropriate technological advancements and socio-technological considerations. Innovations in vehicle automation, control augmentation and display interfaces have enabled pilots with extremely limited flight experience to safely operate a simulated personal aerial vehicle (PAV). Nonetheless, continued efforts are required to make a PATS that can be used by the general public a reality.
The next steps towards a PATS should focus on developing real-world implementations of the automation and augmentation technologies required to bridge the skills gap between a highly-trained pilot a the average car driver. Just as importantly, several key socio-technological issues need to be addressed, including the legal and certification issues surrounding PAV automation and operation.
**Main dissemination activities**
*Scientific publications*
Various scientific publications have been published within the lifetime of the project. These include (but are not limited to):
- 11 journal publications
- 47 conference publications
- 3 articles/sections in an edited book
More publications have been submitted and will be published after the end of the project. An up-to-date list of project publications can be found at http://mycopter.eu/home/publications.html.
*myCopter Project day*
The myCopter Project Day was held on 20 November 2014 at DLR in Braunschweig, Germany. During the event, we presented the outcomes of our project to relevant stakeholders, the general public and members of the press.
In the last 4 years, we have investigated breakthrough technologies in several research areas:
- New concepts for control of PAVs (University of Liverpool)
- Novel human-machine interfaces (Max Planck Institute for Biological Cybernetics, Tübingen)
- Computer vision-based PAV automation (Swiss Federal Institute of Technology Zurich)
- Collision avoidance strategies and automatic landing place assessment (École Polytechnique Fédérale de Lausanne)
- Implementation and test of novel PAV technologies on the DLR experimental helicopter FHS (German Aerospace Center, Braunschweig)
Furthermore, we have explored the potential uses and risks of PAVs for society through technology assessment methodologies (Karlsruhe Institute of Technology).
With scientific presentations and demonstrations we hoped to stimulate lively discussions between attendees during hands-on demonstrations of our findings. Furthermore, attendees of the myCopter Project Day were able to experience demo flights with unmanned aerial vehicles and in DLR’s Air Vehicle Simulator (AVES).
The morning of the myCopter Project Day was devoted to presentations about the scientific results generated within the project. Project coordinator Prof. Dr. Heinrich Bülthoff gave an introduction into the project, after which each project partner gave an overview of their work. These presentations can be downloaded as PDF files from the myCopter website: http://mycopter.eu/home/results/project-day-presentations.html.
After the presentations in the morning, the scientific results were elaborated in various poster prepared by the project partners. During the lunch break, attendees could walk around and discuss the outcomes with the project partners. In-depth posters were presented in the afternoon at each of the demonstrations stands in the exhibition hangar. All posters can be downloaded from the website http://mycopter.eu/home/results/project-day-posters.html.
The afternoon of the myCopter Project Day was devoted to demonstrations concerning the scientific work performed in the project. An overview is given at http://mycopter.eu/home/results/project-day-demonstrations.html. Attendees were first guided along all demonstrations in groups and could explore the entire exhibit at will at the end of the day. These demonstrations featured various simulators highlighting PAV Handling Qualities and human-machine interfaces, Unmanned Aerial vehicles demonstrating vision-based navigation and collision avoidance, videos regarding automatic landing place assessment, and a world café where attendees could discuss their visions for personal aerial transportation systems with project partners.
*Project website, newsletters and flyers*
During the early phases of the project, the myCopter website was transformed from a source of information for the consortium into a more comprehensive resource for the broader community that is interested in enabling technology for PATS. More particularly, we designed and printed information material describing the objectives and expected impact of this project as well as future directions of interest. This material is available to the media via download from the myCopter website.
Throughout the project, we have also published several newsletters:
- Newsletter #1, September 2011
- Newsletter #2, March 2012
- Newsletter #3, November 2012
- Newsletter #4, August 2013
- Newsletter #5, August 2014
These newsletters are available on the project website (http://mycopter.eu/home/downloads.html) and were sent to a mailing list of people that had signed up through the myCopter website. The final newsletter (#6) is expected to be published in March 2015.
Finally, we have also produced 2 project flyers. The first flyer provides details about the aim and strategy of the project, whereas the second flyer highlights its achievements. Also the flyers are available on the project website (http://mycopter.eu/home/downloads.html).
*Articles in the press*
From the outset of the project, myCopter has been experiencing a notable media coverage (including Internet articles, blogs and comments). Some of these were actively triggered by inputs from consortium partners, while others were not (blogs in particular). A comprehensive list is available on the project’s webpage at http://mycopter.eu/home/press-coverage.html which includes (but is not limited to):
- 46 Newspaper articles
- 63 Online articles
- 21 Magazines
- 9 TV items
- 14 Radio items
- 18 Blogs
List of Websites:
www.mycopter.eu
Project coordinator
Prof. Dr. Heinrich Bülthoff
Max Planck Institute for Biological Cybernetics, Tubingen, Germany
heinrich.buelthoff@tuebingen.mpg.de
www.kyb.mpg.de/~hhb
The existing problem of traffic congestion combined with its anticipated growth presents a major challenge for future societal mobility and economic growth. A promising solution that combines the best of air- and ground-based transportation to overcome these problems is to establish a personal aerial transportation system (PATS) based on Personal Aerial Vehicles (PAV).
The myCopter project is a unique integration of social investigation and technological advancement to establish candidate enabling technologies for a PATS. Experts from across Europe have developed advanced technologies for automation, handling qualities and human-machine interfaces for PAVs, and have performed socio-technological evaluations to assess the impact of a PATS on society. The outcomes of this project provide a stepping stone for future endeavours aimed at moving personal transportation into the air.
The myCopter project has demonstrated that a personal aerial transportation system can become a reality, given appropriate technological advancements and socio-technological considerations. Innovations in vehicle automation, control augmentation and display interfaces have enabled pilots with extremely limited flight experience to safely operate a simulated personal aerial vehicle (PAV). Nonetheless, continued efforts are required to make a PATS that can be used by the general public a reality.
The outcomes of the myCopter project identify the next steps towards a PATS. The focus should be on development of real-world implementations of the automation and augmentation technologies required to bridge the skills gap between a highly-trained pilot and the average car driver. Just as importantly, several key socio-technological issues need to be addressed, including the legal and certification issues surrounding PAV automation and operation.
Project Context and Objectives:
The existing problem of traffic congestion combined with its anticipated growth presents a major challenge for future societal mobility and economic growth. A promising solution that combines the best of air- and ground-based transportation to overcome these problems is to establish a personal aerial transportation system (PATS) based on Personal Aerial Vehicles (PAV).
Commuter traffic causes peak loads on ground-based traffic systems, with traffic jams as a result. To alleviate this situation, an idea is to use the airspace for personal transportation. The myCopter project combines the old dream of a flying car with the need for changes in our current transportation system. Besides looking at desirable technological developments for such a PATS, the project partners are especially interested in the associated challenges that are connected with the realisation of such a system from a technology assessment’s point of view.
The aim of myCopter is to determine the social and technical aspects needed to set up a transportation system based on PAVs in today’s society. The project focuses on three research areas: 1) the human-machine interface and training, 2) automation technologies, and 3) the socio-technological environment that surrounds PAVs. MyCopter investigates the idea of a PATS based on small personal aerial vehicles flying short distances between home and the workplace that take off and land vertically and that can be used in densely populated areas.
The interaction between the pilot and a PAV is of crucial importance and depends on the dynamic characteristics of the PAV itself. Therefore, a generic flight dynamics model has been studied that simulates the dynamic behaviour of a PAV. Furthermore, PAVs need an intuitive human-machine interface (HMI). Within myCopter, we investigated the use of a haptic shared control in combination with a highway-in-the-sky display. Such an HMI shows the current flight trajectory in a perspective display and provides pilots with haptic guidance forces on a sidestick based on this trajectory.
Navigation in cluttered environments is crucial for safe flight. PAVs need to be able to find their way, navigate to a destination, avoid obstacles and mid-air collisions with other traffic, and find suitable landing places. These concepts have been studied within the myCopter project by taking novel approaches such as navigation based on vision information from cameras, collision avoidance based on simulations of human crowds and landing place selection based on monocular vision information.
The expectations of future users of PAVs cannot be ignored, as flying vehicles will have a large impact on society – or none at all, if they are not accepted. In the myCopter project, we investigated the technological issues surrounding PAVs, especially regarding the identification of technical and operational challenges. Furthermore, we have performed qualitative research into the expectations of potential users towards a PATS and how society might deal with this new aerial traffic system.
The technologies developed in the myCopter project have been demonstrated on various test facilities. Advances in automation technologies were implemented and tested in unmanned aerial vehicles and prototypes for HMI and training scenarios were investigated on fixed-base simulators. We have also implemented some aspects of these technologies into the Flying Helicopter Simulator, operated by DLR, the German Aerospace Center.
The myCopter project is a unique integration of social investigation and technological advancement to establish candidate enabling technologies for a PATS. Experts from across Europe have developed advanced technologies for automation, handling qualities and human-machine interfaces for PAVs, and have performed socio-technological evaluations to assess the impact of a PATS on society. The outcomes of this project provide a stepping stone for future endeavours aimed at moving personal transportation into the air.
Project Results:
**WP2: Flight simulation/training**
This work package (WP) developed a methodology to assess the handling qualities requirements for Vertical Take-Off and Landing-capable Personal Aerial Vehicles (PAVs). It is anticipated that such a PAV would be flown by a ‘flight-naïve’ pilot who has received less training than is typically received by today’s general aviation private pilots. It was found that the methodology used to determine handling requirements for a PAV could not therefore be based entirely on existing rotary-wing best practice, i.e. the use of highly experienced test pilots in a conventional handling assessment limits the degree to which results apply to the flight-naïve pilot. This WP developed alternative methods to assess the handling qualities requirements of a PAV based on both the subjective and objective analysis of performance and workload of flight-naïve pilots in typical PAV tasks. A highly reconfigurable generic flight dynamics simulation model that was used to validate the methodology was also developed. Results that highlight the efficacy of the various methods have been achieved and their suitability for use with flight naïve pilots demonstrated. The methods developed off the possibility of providing a means to quantify the current subjective methods used for professional pilots.
The WP undertook research to develop handling qualities guidelines and criteria for a PAV. The objective was to identify, for varying levels of flying skill, the response type requirements in order to ensure safe and precise flight. The work has shown that conventional rotorcraft response types such as rate command, attitude hold, and attitude command, attitude hold are unsuitable for the anticipated population of PAV pilots. However, so-called augmented response types such as translational rate command and acceleration command, speed hold permit ‘flight naïve’ pilots to perform demanding tasks within the required precision repeatedly. This is an interesting result as the current standards indicate that a rate command system should be sufficient for professional pilots to achieve the highest level of handling qualities in a good cueing environment but, as the environment degrades to its worst level, a translational rate command is required. PAV pilots require this highest level of vehicle augmentation for the good cueing environment but require no further augmentation as, for instance, the visual scene is degraded.
This WP also undertook research activities to development the training requirements for pilots of PAVs. The work included a Training Needs Analysis (TNA) to determine the skills that would need to be developed by a PAV pilot and the development of a training programme that covers the development of the skills identified by the TNA. The effectiveness of the training programme has been evaluated using the first three Levels of Kirkpatrick’s method. The evaluation showed that the developed training programme was effective, in terms of engaging the trainees with the subject, and in terms of developing the skills required to fly a series of PAV-mission related tasks in a flight simulator. The number of hours training required in a simulator indicates that they would not be prohibitively large when compared to, for example, learning to drive a car.
Motivated by simulator test subjects who expressed discomfort during certain landing profiles that were tested as part of the project, the WP also studied the design and assessment of a "natural feeling" landing profile to guide PAV occupants from cruising flight, down an approach path, to bring the vehicle to a successful hover, in other words, to ‘park’ the vehicle. The development of the new profile was motivated from the point of view that ‘natural-feeling’ cues are related to the physiological cues presented during a visual landing. As such, test subjects with little or no prior flight experience flew simulated approaches to a hover, following limited instruction in the use of a vehicle model. It was found that, despite the wide range of skills and experiences that the participants brought with them to the testing, the approaches that they flew were all broadly similar and could be grouped into three distinct phases. In fact, the approaches heavily resembled those approaches flown by skilled pilots in an earlier NASA study. This previous work in this field, as well as tau theory (a theory from psychophysics that posits that animals in the natural world guide themselves around the cluttered environment that is the Earth’s surface using the direct perception of time-to-contact to an object or surface) was used to design an idealized approach profile based upon the simulation results. It was found that the PAV occupant’s preference for the approach profile depended upon whether or not they were flying the vehicle themselves. The results indicate that the PAV occupants preferred a descent with a constant deceleration profile for an automated landing but for a manual landing, i.e. one where they were flying the vehicle themselves, the designed ‘natural’ profile was preferred.
Finally, this WP also investigated the use of novel control systems designed to provide a safe and reliable method for the control of a future PAV. As it is envisaged that most PAV ‘pilots’ will be familiar with driving a car, the use of response types and controls derived from road vehicles was investigated. Objective and subjective techniques were used to quantify and qualify the applicability of automobile-like response characteristics using traditional helicopter control inceptors modified to behave somewhat like automobile controls, e.g. using foot pedals to control an Acceleration Command, Speed Hold system. Additionally, the effects of eliminating vehicle pitch and roll dynamics were investigated to determine whether this results in a reduction of workload for non-professional pilots. The results suggest that, particularly for the most inexperienced of pilots, the automobile-like configuration is more suitable to allow them to control a PAV than an augmented set of helicopter-style response types. This was shown through increased performance, and a reduction in subjective workload ratings. Improved Handling Qualities Ratings (HQRs) were also obtained in the automobile-like system for the tasks undertaken. Overall, removal of the pitch and roll dynamics was not found to significantly affect task performance in the automobile-like system, but their absence resulted in a decrease in performance for the rotorcraft-style response type configurations tested.
**WP3: Human-Machine Interface for controlling a PAV**
Controlled experimental environments provide the perfect means for developing and testing novel concepts for human-machine interfaces. Within myCopter, we have developed experimental facilities to study pilot behaviour with novel human-machine interfaces in an active control loop. In these experimental facilities we have performed basic research into novel approaches for haptic guidance and human factors (for example, cognitive workload) for personal aerial vehicles. Our studies cover a range of topics such as the design of haptic feedback systems, workload assessment with physiological metrics, looming warning signals and the development and identification of dynamics models of helicopters to aid the implementation of real-world augmentation strategies for non-expert pilots.
*Haptics research*
Our research has led to the development of a prototype human-machine interface (HMI) for personal aerial vehicles that is based on the combination of a haptic shared control framework and a Highway-in-the-Sky display. We investigated whether this combination could result in improved performance for non-expert pilots flying a personal aerial vehicle. Various representations of a flight trajectory in a highway-in-the-sky display were evaluated. It was found that a tunnel and a wall representation led to the best performance, whereas a highway representation resulted in worse performance and higher control activity and effort. Haptic guidance cues on the sidestick allowed pilots to achieve better performance with lower control activity.
The prototype HMI was demonstrated to the general public and the press. Anybody with at least some driving experience could easily get accustomed to the PAV flight dynamics and use the supplied visual display and haptic guidance cues to follow a flight trajectory without large deviations. Participants reported that they found the guidance forces in the various degrees of freedom very intuitive.
Our prototype HMI was based on more fundamental research focusing on the influence of haptic feedback on pilot control behaviour. Two major approaches were identified, the indirect and the direct approach. The indirect approach provides haptic feedback cues that a pilots needs to oppose, while the direct approach provides guidance cues that a pilot needs to follow. Our research has shown that the indirect approach outperforms the direct haptic approach during aircraft pitch angle tracking. This higher performance is due to pilots changing the neuromuscular settings of their arm (admittance) from compliant to stiff when they were provided with indirect force cues as compared to direct guidance cues. However, a disadvantage of the indirect approach is that pilots need to invest more energy into the control task as they have to adopt a “stiff” control strategy by co-contracting the muscles in their arm.
Our novel HMI combines automated control with manual control. The haptic feedback system assists vehicular steering by sharing manual control with the human operator and guiding the operator towards an optimized trajectory. Furthermore, such an approach allows him/her to augment, comply with, or resist the haptic guidance forces according to his/her preferences. Therefore, haptic feedback forces can be used to improve performance and increase safety. Nonetheless, the human operator may not always benefit from the haptic support system. Depending on the amount of haptic feedback, the operator might oppose this haptic assistance or demonstrate an over-reliance. Therefore, we have investigated the levels of haptic assistance that would optimize shared control between the human operator and the vehicle and have shown that pilots can optimise their response in response to the provided haptic support.
*Human Factors research*
Our research has also focused on the human factors associated with PAV flight (e.g. cognitive workload) by employing non-obtrusive real-time methods such as gaze tracking, electro-dermal response, heart-base measures and electro-encephalography (EEG) to estimate the perceived difficulty of PAV flight tasks.
One example of this work is that by tracking the eye movements of pilots in a cockpit environment, we were able to evaluate how humans coordinate their need for visual feedback during flight control. This has allowed us to identify the expected response latencies of the pilot to the appearance of peripheral targets. In addition, we have identified a behavioural feature that distinguishes novice pilots for their control performance: eye-movement efficiency across visual instruments. We have also verified the relative importance of different cockpit instruments in order to complete a PAV scenario. With this in mind, perspective flight-path displays should be designed to emphasize these informational aspects, according to their gaze frequency of on a traditional flight deck.
Another example of our research deals with delays between user input and the system’s reaction in control tasks, which have been shown to have a detrimental effect on performance. This is often accompanied by increases in self-reported workload. We have identified physiological measures that correlate with pilot workload in a PAV that suffered from varying time delays between control input and vehicle response. For this purpose, we measured the skin conductance and heart rate variability during flight manoeuvres in a fixed-base simulator. Control error and the self-reported workload increased with increasing time delay. Skin conductance and input behaviour also reflected corresponding changes. Therefore, these physiological measures are sufficiently robust for evaluating the adverse influence of system delays.
Commonly, the extent of mental workload can be estimated by self-reports (e.g. with the NASA Task Load Index (TLX)). Within myCopter, we have been working towards evaluating workload physiologically in terms of how a primary task taxes a common and limited pool of mental resources, to the extent that it reduces the electroencephalographic (EEG) responses of the brain to a secondary task while performing a primary flying task. Our experiments show that changes in the EEG response of a pilot can be used to identify the control effort required in a specific flight task, but that it is not sufficiently sensitive to provide a graded response across different levels of disturbances. We are still performing further research to improve upon the sensitivity of the EEG metrics.
Finally, we have also investigated the effectiveness of warning cues. Automated collision avoidance systems promise to reduce accidents and relieve the driver from the demands of constant vigilance. Such systems direct the operator’s attention to potentially critical regions of the environment without compromising steering performance. Auditory warning cues with rising intensities are claimed to be especially salient. By evoking the percept of an approaching object, they engage a neural network that supports auditory space perception and attention. Our experiments have shown that we are indeed aroused by and faster to respond to ‘looming’ auditory signals, which also increase heart rate and skin conductance activity.
*Augmentation strategies*
Finally, we have also developed dynamic models of helicopters to aid the implementation of real-world augmentation strategies to aid non-expert pilots. We are investigating the interaction between a pilot with limited flying skills and augmented vehicles that are as easy to fly as cars are to drive (for drivers with limited experience). We focus on light helicopters as these best reflect the properties of a vehicle that could be used in a personal air transport system. Therefore, we are identifying a rigid body dynamic model of a light helicopter in flight tests, such that we can implement augmented system features that allow a pilot with limited flying skills to reach similar performance as a highly-trained pilot. This will be tested in handling qualities and human performance evaluations in piloted closed loop control tasks, with and without the augmented systems.
Similarly, we have developed a non-linear helicopter model for simulating realistic flight scenarios. Commonly, non-linear models are developed for unmanned small-size helicopters, but nonlinear models for full-size helicopters are not very common, due to the difficulty of accurately implementing the components of the vehicle and obtaining reliable aerodynamic parameters. We implemented and validated a model by performing test manoeuvres with the help of closed-loop controllers and a helicopter pilot that flew the model in a simulator. This non-linear helicopter model will allow us to investigate novel non-linear augmentation strategies for supporting inexperienced pilots in PAVs.
**WP4: Control and navigation of a single PAV**
*Vision-aided navigation*
This work package addressed the automation challenge for a single PAV with the aim of developing the intelligence towards autonomous navigation of a Personal Aerial Vehicle (PAV) as defined in the context of the myCopter project. In general, a PAV flight can be divided into three phases: 1) take-off, 2) cruise flight and 3) landing. In order for automation to work in all phases, a good estimate of both the state of the vehicle (position, velocity, orientation) and the environment are essential. During cruise flight, localization of the vehicle can usually be done by GPS, while for take-off and landing, novel camera-based approaches were proposed and developed, as GPS is not a reliable service in close proximity to urban structure.
We investigated flight-path planning and a navigation display for use during cruise, which resulted in a so-called “Highway In The Sky” (HITS) Display: Conventional helicopter cockpits feature an instrument panel with a variety of instruments that the pilot has to check continuously. The pilot has to connect the individual pieces of information in his mind in order to get a sense of the behaviour of his vehicle. Training and experience is required for effectively piloting a helicopter with the information from a conventional instrument panel only. For PAV pilots who will have only a minimal training compared to today’s helicopter pilots, a more intuitively understood instrument panel must be designed. A strong candidate for this purpose is the highway-in-the-sky (HITS) display. The HITS display is an easy to use navigation display. A tunnel depicts the flight trajectory. The tunnel shape can be selected to represent walls, a u-shape, a box or a highway. Overlaid to the three-dimensional design is a two-dimensional primary flight display (PFD). This features conventional indications for attitude, heading, altitude, airspeed and torque. Additionally, optional target indicators, so called bugs, are implemented for flight parameters such as airspeed and altitude as well as pitch angle and torque. These bugs have a rectangular shape with cut-out triangle in order to provide the pilot with information on the acceptable deviation from the target value.
These phases for take-off and landing are considered to take place in urban environments, where reliable GPS service cannot be guaranteed anymore. We have used cameras in combination with an Inertial Measurement Unit (IMU) as the main sensor setup for localization in the control loop. We identified perception and state estimation as the key components that enable autonomous or automated flights in GPS-restricted areas. Only if these components provide a good estimate of the current state of the vehicle, controllers stabilizing the vehicle can work reliably and fly to waypoints or tack trajectories created by a path-planner. Using camera(s) as the only exteroceptive sensor(s), we fuse inertial measurements to achieve a self-calibrating power-on-and-go system, able to perform autonomous flights in previously unknown, outdoor spaces. Our framework achieves Simultaneous Localization And Mapping (SLAM) with previously unseen robustness in onboard aerial navigation for small platforms with natural restrictions on weight and computational power. Successful navigation and control of this real system has been thoroughly tested in real flight experiments, and under a variety of challenging conditions. All flights were performed autonomously using visual cues directly in the control loop, with only high-level waypoints given as directions. The state estimation framework was furthermore improved to be able to process delayed, relative and absolute measurements from a theoretically unlimited number of different sensors and sensor types, while allowing self-calibration of the sensor-suite online. This work done in myCopter was amongst the first that showed successful navigation of a Micro Aerial Vehicle (MAV), solely based on visual and inertial measurements and their fusion.
As visual-inertial based localization and control require certain excitation in order to render all their states observable, we went one step further and developed a path planning method, taking these requirements into account. The method also incorporates the ability of the cameras to localize reliably and plans safe paths to the selected goal. That is, we seek to find not only a short path to the destination, but an informative path where the states monitored in our system are best observable at all times, and where the goal can be reached within some given confidence area. As an example, it is safer to fly a small detour, in case a direct path to a landing place does not exhibit sufficient visual features.
In order to be able to follow the paths generated by our planning method, we have developed a simple but powerful control approach for the MAV, which can be easily deployed on computationally restricted embedded hardware, and increases bandwidth. This controller is kept generic in order to be applicable to a variety of rotorcraft.
In summary, we have developed novel methods for control and navigation of aerial vehicles. These methods were demonstrated successfully on both the simulation facilities as well as Micro Aerial Vehicles. MAVs are great research platforms for performing experiments, and are believed to have properties similar to a future envisioned PAV. Even if not immediately deployed on a PAV, many of the developed methods are already applicable to current systems.
*Landing place selection and assessment *
When it comes to the transition from cruise to landing, we have investigated methods for automated visual landing place selection and assessment in man-made environments. We have developed an approach for automated landing place assessment in man-made environments. With our approach candidate landing sites are detected as featureless, regularly shaped areas in the image that typically characterize man-made landing structures and we devise an algorithm to compute them efficiently. Landing sites are then selected by searching for cycles in a line fragment graph formed from lines detected about the contours of candidate image regions, using learned measures of object appearance and geometry.
Our approach for candidate landing site detection is based on finding Maximally Stable Extremal Regions (MSERs) that typically correspond to dominant, featureless regions in the image. They are computed by searching for stable regions across varying image thresholds, which for reasonably sized images can be done very efficiently. We further apply Hough line voting to measure the shape of each MSER and detect MSERs exhibiting a polygonal shape. We have demonstrated this approach for the detection of both prepared and unprepared landing sites including airport runways, building rooftops and rural fields, where it was shown to obtain near real-time performance in the case of single frame low-resolution images. A paper detailing this work has been published in the Machine Vision and Applications journal. We have also developed a multi-feature version of our algorithm and demonstrated it for landing site detection on sequences captured from the FHS helicopter.
Our algorithm for landing site selection is based on finding cycles in a line fragment graph. More specifically, given the detected MSER regions and line fragments we construct a weighted graph, and search for contour regions as cycles in this graph using a learned cycle scoring function that can be used to select landing sites of a particular type. Importantly, unlike previous graph cycle object detection methods, our approach can be used with a wider variety of cycle scoring functions and image features to result in an increased accuracy. A paper detailing this work has been published in the European Conference on Computer Vision.
Our work on vision-based geographic localization has focused on algorithms for fast and accurate image matching and indexing in large environments. Toward this end, we have developed a boosted binary keypoint descriptor that is not only accurate to match but also highly efficient to compare and store.
In our initial work we proposed a boosted floating-point descriptor (FPBoost), also referred to as Low-Dimensional Boosted Gradient Maps (L-BGM), that applies boosting to learn a visual similarity measure from labelled pairs of similar and dis-similar image patches and optimize over the pooling configuration of the resulting image descriptor. FPBoost was shown to rival the performance of the state-of-the-art floating-point descriptors and provide a significant improvement over SIFT. We then extended this idea to develop a boosted binary keypoint descriptor (BinBoost) that greedily optimizes over a set of binary hashing functions to result in a descriptor that is not only accurate to match but also very fast to compare and efficient to store. BinBoost provides a similar accuracy to the state-of-the-art floating-point descriptors at a fraction of their matching cost, and results in an increased accuracy over the state-of-the-art binary approaches.
**WP5: Navigation in the air and the interaction with other traffic**
*Collision avoidance *
In this work package, we tackled the problem of mid-air collision avoidance in dense aerial environments occurring in the myCopter scenario. High densities of agents decrease the time to react before a collision and thus need a quick response from the agent, limiting negotiation time resulting from collaborative strategies. Furthermore, finite communication bandwidth limits information exchange between agents, leading to an incomplete view of the environment. A reliable collision avoidance strategy for high densities traffic should therefore be reactive, fast and robust to unforeseen circumstances. Centralized strategies suffer from a single point of failure and bottlenecks. The need of constant recalculation and time to transmit solutions prevents the use of such methods in these dynamic and dense environments.
With the use of local sensing and finite range communication devices, the knowledge of the environment is incomplete and global path planning methods would therefore potentially fail by not considering a future threat that they are not aware of at the time of planning. Reactive approaches where the controller adapts to the actual situation are more suited to address this problem.
With passengers on board, the question of comfort during the flight has to be addressed. The commuting journey should be as smooth as possible and not exceed physiological comfort limits.
To alleviate these challenges, we proposed a crowd-inspired, reactive approach to address collision avoidance in dense aerial traffic, and developed an extension to improve passenger comfort. Extensive simulations have been carried out with more than 200 simulated PAVs to evaluate travel time and relative comfort. Additionally, we show that the proposed strategy can be implemented on unmanned flying robots in real-time with minimal computing requirements in an experiment with ten robots flying outdoors. We developed the entire platform used to perform these tests. To facilitate safe experimentation with many robots, the Micro Aerial Vehicle was designed to be very lightweight (<500 grams), very easy to assemble and repair, and low cost. We have now a fully working quadrotor and autopilot, which is easy to build based on 3D-printed parts and low-cost off-the-shelf components, with full access to the software. The software and the hardware components will be open-sourced.
In addition, we showed how this framework could be extended to deal with non-flying zones. Non-flying zones are geographically defined zones were no flying vehicles should overfly. Cities’ downtown, large airports, nuclear plants and military basis are examples of places that no one is allowed to fly over.
Finally, we demonstrated how the standard Reynolds flocking rules could be modified in order to deal with agents with individual goals. Evolution came up with a simple solution to navigate safely in dense environments. In Nature, flocks of birds, school of fishes, swarms of insects, human crowds and bacteria are examples of collective behaviours emerging from local control and local sensing. It can be observed that large-scale behaviours coming from self-organization process emerge spontaneously from only local interaction and local sensing. Reynolds proposed some simple rules to describe the flocking of birds. However, these rules apply to agents having a common goal such as migration. Therefore, with standard Reynolds flocking, the agents would have to alter their trajectory too much in order to stay with the flock. On the other hand, with our solution, flocks of PAVs will be naturally joined and left depending on the flight direction of each PAV. With this framework, the advantages of spatial closeness and the egoistic behaviour of PAVs will be optimally combined to guarantee a safe and direct navigation to each one’s goal.
*Vision-based detection of other PAVs*
We developed approaches for the detection of both collision course and general flight of neighbouring PAVs. To detect PAVs on a collision course, we not only use their distinctive appearance but also their unique motion pattern. Toward this end, we employ a spatio-temporal Histogram-of-Oriented Gradients (HOG) feature along with a linear Support Vector Machine (SVM), evaluated using a sliding window approach over multiple spatial window sizes. PAVs on a collision path are then identified as those that appear stationary and growing in size. We evaluated our approach using a video dataset. Exploiting the temporal pattern of aircraft on a collision course results in a highly accurate performance that significantly outperforms 2D detection with or without temporal smoothing. When additional computational resources are available an even further increase in accuracy can be achieved using larger window sizes.
Under general flight both the motion of the aircraft and the neighbouring aircrafts must be compensated. We first compensate for camera motion using local feature matching. Neighbouring aircraft are then detected using a sliding-window linear SVM classifier with spatio-temporal HOG features, where the frames of each spatio-temporal window are normalized according to the motion of the neighbouring aircraft, whose motion is estimated using a learned mapping function. Our approach achieves a significant improvement over 2D detection alone with a 2D HOG classifier or boosted trees, the comparison with not aligned 3D HOG highlighting the importance of aircraft motion correction.
Both of our approaches for the detection of neighbouring aircrafts require annotated aircraft images that can be costly to collect. We therefore developed a synthetic data generation approach to help alleviate manual annotation effort. Provided a 3D model of an aircraft, for each annotated aircraft image we generate synthetic images by rendering the model and matching the rendered aircraft image to the real one by optimizing over its rendering parameters. Our approach results in a significant improvement over labelled-only performance, the most significant improvement resulting when using the supervised image matching criteria of the corresponding classification scheme.
**WP6: PAV operational system concepts**
The objective of this work package (WP) was to develop the concept of a vehicle and the supporting systems that will fit the needs of the overall call for a PAV.
The first achievement of this WP was the development and analysis of design features and options for a PAV and the supporting system or infrastructure. This started with descriptions of the intended use case scenarios for PAVs and the intended users. PAVs are foreseen to be commuter vehicles in the first place. They shall provide point-to-point connections for typical commuting distances and shall be competitive to existing ground-based transportation methods. PAVs are supposed to be available to the general public, which leads to new requirements regarding the qualification of pilots. Obtaining a PAV license must become clearly less complex in terms of necessary knowledge, training time and financial investment than keeping a helicopter pilot license today (e.g. PPL-H).
The consortium agreed upon initial design features as a common basis for further development of a reference PAV. A preliminary design study based on helicopter theory methods and experience from pre-design of conventional helicopters was conducted (e.g. power consumption calculations were used for estimating necessary battery capacity and weight). Two development phases have been elaborated regarding the necessary level of automation, which cover different automation functionalities. During the first phase PAVs would be equipped with certain assistance functions, e.g. for stabilization and altitude, speed control or GPS navigation, but the PAV user would still need to fly manually in certain flight phases. The second phase would then assume to have fully automated PAVs. The user would not be pilot anymore but could be understood as human payload in a fully autonomous air vehicle.
This WP also undertook research in the field of a novel easy-to-handle human-machine interface suitable for future PAV pilots. The control concept of automobiles is well known to the general public and the related inceptors have barely changed over the past century. Although new controllers like joysticks are technically feasible, all of the modern production vehicles rely on the conventional arrangement of steering wheel, accelerator and brake pedals, gear stick, and optionally clutch pedal for manual transmission. A driver’s license holder can intuitively connect the usage of these controls to the movement of his car. On the other hand, conventional helicopter controls are not at all intuitive for non-expert pilots. There exist PAV prototypes that feature different steering concepts for different phases of operation, such as a steering wheel for ground operation and conventional gyrocopter inceptors for flight. This means that the user must learn to use two different control concepts and even switch the inceptors for transition from road to air mode. The approach within this project was different and our goal was to simplify usage instead of multiplying the amount of inceptors. The average car driver has gained extensive experience in steering wheel control of ground vehicles. We wanted to make use of this prior knowledge and extensive amount of training to transfer the benefits to rotorcraft control. Therefore, the intuitively understood steering wheel was used as central element of the new rotorcraft control concept. Together with the advanced response types for PAVs developed in WP2, a car-like steering concept for rotorcraft was developed and tested.
These PAV technologies were integrated and tested in a motion-base simulator. This allowed us to perform handling qualities evaluations under conditions close to real flight. Also certification activities were undertaken such that the novel steering wheel concept for PAVs could be tested in-flight. Unfortunately, a technical failure occurred on the basis aircraft that grounded the research helicopter until the end of the myCopter project, which prevented us from pursuing a full flight test regime, but flight tests are still planned once the research helicopter is back in operation.
The baseline configuration for the investigations in the simulator flight trials was the hybrid flight dynamics PAV model with different characteristics dependent on the flight condition, as developed in work package 2. Five pilots participated in a handling qualities study where the four mission task elements had to be completed (Hover, Hovering Turn, Vertical Manoeuvre and Slalom). The baseline configuration with conventional pilot controls received average handling qualities ratings (HQR) in the desirable Level 1 regime. The baseline configuration was then compared to a configuration with activated model-based control (MBC). This control technique was needed for in-flight simulation on the research helicopter. A quantitative analysis showed the model following capabilities to be acceptable to good. Deficiencies were found in the hover where drifts occurred. Evaluations with five pilots showed that the MBC configuration was flyable with handling qualities ratings in the Level 2 regime (adequate).
Finally, a steering wheel inceptor has been implemented as the primary inceptor for PAV control. The steering wheel replaces the conventional centre helicopter stick, which is a novelty in rotorcraft control. The control axes are re-defined to resemble a car-like control concept. Thus, the steering wheel is used to command a combination of yaw and roll inputs depending on the current airspeed. The pedals are used for controlling acceleration and deceleration along the longitudinal axis. The collective lever is used for height control. An additional 8-way switch in the centre of the wheel features precision manoeuvring functions. In the piloted evaluations the awarded handling qualities ratings for all mission task elements were better for the steering wheel configuration compared to the configuration with conventional inceptors. The workload, which was measured by the NASA Task Load Index (TLX), was also lower when the steering wheel was used. The only exception is the hovering turn. Here, the workload slightly increased due to the new sequential control strategy the pilots had to adopt. Although the pilots involved in the study rated the steering wheel configuration with a lower overall score on the System Usability Scale, the better HQR and TLX ratings prove the feasibility of a steering wheel control concept for rotorcraft, which was envisioned to be especially suitable for the pilots of future rotorcraft with highly augmented control laws such as PAVs.
**WP7: Exploring the socio-technological environment**
The main goal of this work package was dedicated to gaining deeper insights into the socio-technological context and the infrastructural environment of a future personal air transportation system (PATS). The operation of personal air vehicles (PAVs) raises plenty of questions about their potential impacts on society, and it is not clear what the expectations of society regarding PAVs actually are and how they will be met. Since PATS are in a very early stage of development, the demand for this form of transportation is still vague, as are people’s preferences for the design of PAVs and their associated infrastructure. It is also imaginable that a broad implementation of PATS may, at least potentially, have significant broader implications for a complex socio-technical system: mobility.
In order to explore these issues, at first a scoping phase was conducted with the aim to map the socio-economic environment of this new transport form, to identify the challenges and issues surrounding an actual realization of a PATS and its integration into the existing transportation system in Europe (see Deliverable D7.1 of the myCopter project). The potential opportunities, obstacles and challenges of PAVs were identified by an extensive literature research on topics such as technologies, regulation, safety, ATM, UAVs etc., supplemented by expert interviews. The key issues found to be relevant for a real world implementation of PATS into our current cities were safety concerns, including the weather issue, legal aspects (certification, insurance, airspace regulation etc.), technical and operational challenges such as automation and autonomy issues, Air Traffic Management (ATM), parking, storing and maintenance infrastructure, and socio-economic and ecological challenges such as energy and noise, acceptance and the aspects of system integration.
One of the key ideas of myCopter is to explore the potential of PATS/PAV to open the third dimension for commuter traffic and thus to reduce the negative impact of congestion in European metro areas. A general assessment of the overall impact of PATS on European road traffic appeared to be difficult for methodical reasons. The regional impact of PATS may differ considerably, depending on the respective local situation, since all cities/metro regions differ with regards to a number of factors such as spatial structure, structure of the local economy, socio-demographics, and many more. In general, one could imagine conducting site-specific studies for all relevant metro areas within the EU. But this would be far too complex for a single research project – and most likely also too early, given all the other uncertainties mentioned above. Therefore, it was decided to follow a more generic approach: Short narrative travel scenarios, describing stylized trips to and from work (addressing different settlement densities and architectures as well as autonomy levels/user qualifications) were developed and used to find categories of requirements which should be discussed and framed in order to clarify the vision and to develop a coherent “reference case” for the vehicle.
Coherence, here, is based on the understanding that the design parameters in the reference case cannot be set independently. A decision for one performance requirement, for example the seating capacity of the PAV, has an influence on other requirements of the PAV (internal dependency) like take-off weight. Some requirements are also strongly connected to the mission that the PAV shall provide for.For instance, cruising speed for PAVs has to be significantly higher than for cars in order to offset time losses during pre- and post-trip procedures and to remain competitive with regards to door-to-door travel times. With regards to vehicle size, it was decided to go for an option to transport either one person with some luggage located on the second seat or to provide transport for two persons – which reflects today’s realities in occupancy rates in commuting. This, in turn, permitted to reasonably assume a take-off weight of 450 kg. Calculations by the project partner DLR showed that PAVs of that size could be operated with a fully electric propulsion system, using either next stage batteries or fuel cells, and provide for a satisfying range.
As the PAV in commuting is competing with private cars and should be capable of being integrated into the existing transportation system, a car-like physical dimension was seen as wise in order to be able to use existing infrastructures for parking and more. As the Reference PAV is assumed to have VTOL abilities in order to be able to start and finish trips close to homes or in CBDs, respectively, only limited manoeuvrability - in order to enable access to storage facilities - but no “roadability” was foreseen.
Considerations on weather dependence started with the premise that, in the context of commuting, a PATS has to be available year-round – at least in principle. On the other hand, experience shows that this is virtually impossible to achieve, at least in most regions in Western and Central Europe. On order to illustrate the related challenges, a benchmark of 90 % usability year-round was deliberately set. This figure is well below the estimates for the usability of cars in the same region but still very ambitious for a PAV. Assessments based on General Aviation Forecast (GAFOR) data for Rhine-Main showed that this could only be reached if for some trips quite bad weather conditions with a visibility on the ground of less than 1.5 km and/or a cloud base height of less than 500 ft would be accepted by the user and could be safely provided by the system. These results illustrate that the topic of how to expand the operability of the PAVs into challenging weather conditions will have to be addressed further if PAVs should become a serious solution for everyday mobility. They also underscore that PAVs, in order to become a meaningfully usable part of everyday transportation, need to be technically able to provide more options than just those following visual flight rules (VFR).
In a similar manner, heuristics for a commuting scenario using a “stylized” European city were developed. It was shown that, assuming a number of approx. 300,000 people that commute every day into a major city, modal shares typical for European cities and a net substitution rate of 10 % of car traffic by PAV, an “automated” ATM for such a prototypical city would have to handle between 2,500 and 10,000 approaches per hour. Between 40 and 160 independent landing sites for PAV would be needed. Further assuming a conventional business model (“individual ownership”) and limited autonomy (no ability of fully automated flying) of PAV, this scenario indicates a required storage capacity for 7.000 to 20.000 PAV within the city. Especially the required capacity for storing temporarily unused PAV in a city centre would, in our opinion, substantially limit the diffusion potential of this technology. In other words, it would – among other reasons – support an implementation strategy that builds on fully autonomous “PAV on demand” which, like a driverless taxi, is able to pick up riders at home and drop them off at a number of dedicated sites within a city.
In order to do explore and understand the perspectives and expectations of potential users and/or citizens with regards to PATS development, three focus group interviews with laypeople were performed in the myCopter project and their outcomes were used to revisit the PATS concept as well as the specifications of the reference PAV (see Deliverable D7.3). The general technical feasibility of a PATS was not a major issue in all of the focus groups. Main issues identified in desktop research reappeared in the focus group discussions (like environmental impact, especially noise footprint and ‘fuels’, driver education, automation, availability, parking …) and were, in a way “hardened”. It was, for instance, strongly confirmed that the environmental footprint in general and in particular the energy consumption of the PAV must be as low and as renewable as possible. Therefore, the strong request for an electric propulsion concept for a PAV was further supported. Nonetheless, even after a lively debate about potential advantages, also doubts about the actual benefits remained among a number of the participants. A frequently discussed question was whether PATS would really improve the traffic situation on the ground or if every newly created mobility option and added capacity would not just lead to new traffic up to a point where everything – including the air space – would be overcrowded again. This mirrors a quite prominent but still controversial discussion in transportation planning, according to which many (urban) roads have latent travel demand and additional vehicle trips will occur if congestion is relieved – a rebound effect of infrastructure expansion also known as “induced travel”. Some perceived PAVs as “over-engineering”, stating that the level of automation needed to have PAVs would probably already solve congestions problem on the ground if implemented in cars.
But also new, additional aspects were mentioned that could not be investigated further in our study but might be subject to further considerations, like the usability of PAV “on the ground” (flying at 20 cm level), e.g. to integrate them in conventional road traffic under certain circumstances, since at least during a transformation period not all destinations could and would be equipped with dedicated infrastructures for PAV and hence a “dual use” of existing infrastructures both for PAV and cars should be considered. This would also address some concerns about the limitations to spontaneity and to integration into ‘real life’ trip chains that a PATS might incur. Beyond that, “more simple” options for early implementations were proposed. One idea was to use PAV like “rope-less aerial ropeways” – point-to-point connections in geographically challenging or already overcrowded areas where other options that rely on extensive fixed infrastructure are not technically and/or economically feasible. They would leave the technical system in the hands of professional operators (and reduce non-technical challenges like safeguarding proper maintenance challenges), would allow for testing PAVs in a controlled environment and solve pressing local problems. Comparable arguments may hold true for similar applications like island shuttles or valley hopping.
The greatest remaining challenge, nonetheless, is to decide about the automation concept of PAV and PATS. In myCopter, two basic concepts have been considered in parallel: fully autonomous and semi-autonomous (or colloquially: the elevator model and the driving pilot model). Fully autonomous PAVs were thought to perform all flying tasks without any interactions with the user. They would be able to move even without a person on board, allowing to disconnect pre- and post-trip processes from the user and to support new forms of mobility. The idea of semi-autonomous PAVs tries to link today’s routines of driving a car to a vehicle that is capable of traveling in three dimensions. In this case, assistive technology would be implemented on a vehicle that supports the driver in having longitudinal, lateral and vertical control while at the same assisting him in remaining in control, finding directions and avoiding collisions. Both concepts appear to be attractive to potential users, and even the focus group participants were, in a way, undecided about their likings. It was broad consensus that the preference for the degree of automation depends on the purpose of the trip, favouring full autonomy for daily routine trips (like commuting) and the opportunity to use a PAV more freely during leisure trips or “just for fun”.
The “automation question” was also a central topic of a subsequent expert workshop with academics, pilots and interest club members. Here, especially the challenges of designing semi-autonomous PAVs, their HMI and potential development paths towards full autonomy were discussed in greater depth. The experts argued that if the user would be foreseen (by the developers) to be responsible for taking over flight tasks in case of an unexpected situation, or a situation no longer manageable by the system, this would mean that the user would have a function like a professional safety pilot. They were very critical about a development philosophy that relies on the user as this kind of emergency backup. This role would be challenging even for well-trained pilots and obviously contradict the development goal of limited training requirements for a general PAV user. Instead, experts recommended considering a very communicative and transparent system which continuously informs the user about its current state as well as its intentions. For the semi-autonomous mode they suggested to permanently keep the user “in the loop” to avoid long delays before regaining full situation awareness.
This work package of the myCopter project has succeeded in providing first insights into the interdependence of the various factors in PAV/PATS development and realization but still was largely explorative. Future system studies will have to focus on a PATS concept with a further reduced option space. What can be said, nonetheless, is that the implemented level of autonomy of the vehicles will be a key design feature and decisive for future deployment strategies (and, in consequence, of more detailed assessment scenarios).
Potential Impact:
**Potential impact**
Several issues need to be resolved before a personal aerial transportation system can be realised in today’s society. The myCopter project aimed to make specific advancements in the user-centred design of human-machine interfaces and training for personal aviation, automation of personal aerial vehicles (PAVs), and the socio-technological assessment of the impact of a personal aerial transportation system on society.
Within the project we have evaluated several enabling technologies. Novel approaches to human-machine interfaces, such as haptic shared control and highway-in-the-sky displays, have been developed for use by non-expert pilots in highly augmented PAVs. We have shown that this combination can provide non-expert pilots with an easy-to-use control interface for flying a personal aerial vehicle.
We have also implemented a steering wheel in a helicopter that replaces the conventional centre stick. The control axes are re-defined to resemble a car-like control concept. Thus, the steering wheel is used to command a combination of yaw and roll inputs depending on the current airspeed. The pedals are used for controlling the longitudinal axis. The collective lever is used for height control and an additional 8-way switch in the centre of the wheel features precision manoeuvring functions. Experimental evaluations have shown that this concept a good alternative to conventional helicopter controls as pilot workload was decreased and handling qualities ratings increased in PAV flight tasks.
Furthermore, we have developed requirements for the Handling Qualities of future PAVS, together with a training syllabus for prospective PAV pilots. A training needs analysis for PAV flight resulted in the definition of five ‘lessons’, each focusing on a specific part of the PAV flight envelope and specific PAV manoeuvres. Several volunteer student PAV pilots have been trained for PAV flight with the training syllabus. They could complete the training in less than 5 hours on average, which highlights the efficacy of the developed training syllabus and PAV Handling Qualities.
Our approaches for navigation strategies for PAVs have shown that it is possible to reliably use vision-based information for navigating through cluttered environments. Also, it is possible to select landing places from monocular camera information with higher precision than with other methods. Collision avoidance can be performed in high-density environments for agents with individual goals by using strategies based on human crowd simulations.
We have identified user expectations towards a personal aerial transportation system and potential scenarios for PAVs. Through technology assessment methodologies, we offered an integrated view on the socio-technological environment that surrounds the personal aerial transportation system and highlighted pathways for bringing the implementation of a PATS forward.
The partners in the myCopter project believe that the outcomes of the projects will allow taking first steps towards the implementation of a personal aerial transportation system in today’s traffic system. Such a system could have a beneficial impact on our daily lives and could help solving issues related to current traffic and environmental problems. It would allow the possibility of direct travel between two points and therefore avoid people to sit in traffic queues, reducing travelling time and frustration. This could potentially save fuel and would reduce the stop and go traffic that wastes large amounts of energy. Travelling conveniently in the third dimension from door-to-door might allow for less congestion and would not require maintenance of support structures such as roads currently necessary for mass transit.
The myCopter project has demonstrated that a personal aerial transportation system can become a reality, given appropriate technological advancements and socio-technological considerations. Innovations in vehicle automation, control augmentation and display interfaces have enabled pilots with extremely limited flight experience to safely operate a simulated personal aerial vehicle (PAV). Nonetheless, continued efforts are required to make a PATS that can be used by the general public a reality.
The next steps towards a PATS should focus on developing real-world implementations of the automation and augmentation technologies required to bridge the skills gap between a highly-trained pilot a the average car driver. Just as importantly, several key socio-technological issues need to be addressed, including the legal and certification issues surrounding PAV automation and operation.
**Main dissemination activities**
*Scientific publications*
Various scientific publications have been published within the lifetime of the project. These include (but are not limited to):
- 11 journal publications
- 47 conference publications
- 3 articles/sections in an edited book
More publications have been submitted and will be published after the end of the project. An up-to-date list of project publications can be found at http://mycopter.eu/home/publications.html.
*myCopter Project day*
The myCopter Project Day was held on 20 November 2014 at DLR in Braunschweig, Germany. During the event, we presented the outcomes of our project to relevant stakeholders, the general public and members of the press.
In the last 4 years, we have investigated breakthrough technologies in several research areas:
- New concepts for control of PAVs (University of Liverpool)
- Novel human-machine interfaces (Max Planck Institute for Biological Cybernetics, Tübingen)
- Computer vision-based PAV automation (Swiss Federal Institute of Technology Zurich)
- Collision avoidance strategies and automatic landing place assessment (École Polytechnique Fédérale de Lausanne)
- Implementation and test of novel PAV technologies on the DLR experimental helicopter FHS (German Aerospace Center, Braunschweig)
Furthermore, we have explored the potential uses and risks of PAVs for society through technology assessment methodologies (Karlsruhe Institute of Technology).
With scientific presentations and demonstrations we hoped to stimulate lively discussions between attendees during hands-on demonstrations of our findings. Furthermore, attendees of the myCopter Project Day were able to experience demo flights with unmanned aerial vehicles and in DLR’s Air Vehicle Simulator (AVES).
The morning of the myCopter Project Day was devoted to presentations about the scientific results generated within the project. Project coordinator Prof. Dr. Heinrich Bülthoff gave an introduction into the project, after which each project partner gave an overview of their work. These presentations can be downloaded as PDF files from the myCopter website: http://mycopter.eu/home/results/project-day-presentations.html.
After the presentations in the morning, the scientific results were elaborated in various poster prepared by the project partners. During the lunch break, attendees could walk around and discuss the outcomes with the project partners. In-depth posters were presented in the afternoon at each of the demonstrations stands in the exhibition hangar. All posters can be downloaded from the website http://mycopter.eu/home/results/project-day-posters.html.
The afternoon of the myCopter Project Day was devoted to demonstrations concerning the scientific work performed in the project. An overview is given at http://mycopter.eu/home/results/project-day-demonstrations.html. Attendees were first guided along all demonstrations in groups and could explore the entire exhibit at will at the end of the day. These demonstrations featured various simulators highlighting PAV Handling Qualities and human-machine interfaces, Unmanned Aerial vehicles demonstrating vision-based navigation and collision avoidance, videos regarding automatic landing place assessment, and a world café where attendees could discuss their visions for personal aerial transportation systems with project partners.
*Project website, newsletters and flyers*
During the early phases of the project, the myCopter website was transformed from a source of information for the consortium into a more comprehensive resource for the broader community that is interested in enabling technology for PATS. More particularly, we designed and printed information material describing the objectives and expected impact of this project as well as future directions of interest. This material is available to the media via download from the myCopter website.
Throughout the project, we have also published several newsletters:
- Newsletter #1, September 2011
- Newsletter #2, March 2012
- Newsletter #3, November 2012
- Newsletter #4, August 2013
- Newsletter #5, August 2014
These newsletters are available on the project website (http://mycopter.eu/home/downloads.html) and were sent to a mailing list of people that had signed up through the myCopter website. The final newsletter (#6) is expected to be published in March 2015.
Finally, we have also produced 2 project flyers. The first flyer provides details about the aim and strategy of the project, whereas the second flyer highlights its achievements. Also the flyers are available on the project website (http://mycopter.eu/home/downloads.html).
*Articles in the press*
From the outset of the project, myCopter has been experiencing a notable media coverage (including Internet articles, blogs and comments). Some of these were actively triggered by inputs from consortium partners, while others were not (blogs in particular). A comprehensive list is available on the project’s webpage at http://mycopter.eu/home/press-coverage.html which includes (but is not limited to):
- 46 Newspaper articles
- 63 Online articles
- 21 Magazines
- 9 TV items
- 14 Radio items
- 18 Blogs
List of Websites:
www.mycopter.eu
Project coordinator
Prof. Dr. Heinrich Bülthoff
Max Planck Institute for Biological Cybernetics, Tubingen, Germany
heinrich.buelthoff@tuebingen.mpg.de
www.kyb.mpg.de/~hhb
