Multi User Virtual Interactive Interface

The MUVII project has achieved something very new in its haptic feedback system development. It has combined force feedback with tactile feedback. All of today's force feedback systems are based on the premise of a single feedback vector (directional) force. For enhanced virtual reality the MUVII haptic system added "surface detail" information, such as the "angle" or contour of the touched point which the user's finger experiences in the real world are also made available through the active surface of the tactile motor. The Immersion company has attempted to simulate the sensation of "touch" in their haptic glove by providing an impulse to the finger nail when the finger tip pulp comes into contact with the virtual object. MUVII not only provides this information directly to the fingertip pulp, but also the angular contact and even texture information. The MUVII system was first demonstrated to the general public at Laval Technopole in May 2004. Please refer to the written user response reports "IKD and ITD comments_LV04.doc" where many users specifically commented on the new tactile sensations when touching objects, and specifically when feeling the rotating gears. To the best of knowledge all of today's force feedback systems are based on a starting or datum point. In other words, all these systems work on the assumption based on object and surface coordinates are measured from a single fixed datum point. What is new in the ITD system is that you can move your arm around in a large 3D room and when your hand intersects with an object it is a relative calculation and it is just the wrist which feels the obstruction, your arm is still "floating". Another way to view this is the way we use the computer mouse, which also is a relative device. Then think of a mouse programmed as a datum device such that the mouse pad exactly represents the screen area, and the position of the mouse on the mouse pad corresponds directly to the position of the cursor on the screen. That is the way most/all force feedback devices are programmed.

The new algorithm IKA-PAN was developed to pan an audio signal between elevated loudspeakers, which means it generates different signals for the loudspeakers that the sound event appears clearly in between the elevated loudspeakers. With this algorithm the 3D-sound reproduction over a loudspeaker array is improved in either reducing the number of necessary loudspeakers for a given reproduction quality or by improving the reproduction quality keeping the number of loudspeakers, which were used without this algorithm. The new algorithm enlarges also the reproduction area.

When we started the MUVII project, we knew that the task would be difficult. Indeed the haptics technologies were not much developed until this day because of their cost and the difficulties of integration. In the current state of technology and market, it is necessary to notice that the existing products are American or Japanese, that they are not portable and that they are very expensive. The idea to move from the notion of return-of-effort to that of return-of-sensation in order to get a light device at a small cost attracted the MUVII project partners but also the end users (e.g. theme parks, museums, video games producers and device manufacturers for general public market). We have demonstrated that we can produce in a reasonable period and at an acceptable cost, three functional industrialized prototypes (ITDx2 and IKDx1). We acquired this certainty by making the strategic choice to simplify the devices. This choice allows significantly reducing the cost of reproduction of the prototypes.

Thanks to the CEA LIST, extensive knowledge of haptics, specific robotics design methodology, and dimensioning and optimisation tools, a major step was covered towards the development of intuitive and effective wearable haptic devices, with a variety of new potential commercial applications. In particular, an innovative and patent pending 2DOF wearable haptic interface with force and tactile feedback was developed. It allows power grasp intuitive manipulation of virtual tools. Wireless 6DOF tracking allows free hand movements (range of motion around wrist axes exceeds 70° while the device can be freely moved over more than 1 meter). High-end actuation stage enables more than 5N high quality force feedback in hand, rendering realistic interactions between the virtual tools and the environment. Moreover, tactile actuators also allow shape and textures exploration and recognition. These breakthroughs compared with traditional pointing devices like mice or joysticks allowing basic 2DOF target control on screen without feedback enable a better immersion of users in a virtual world, as they become real actors. This was verified with cultural group interaction applications in an Immersive Theatre Demonstrator. This could be applied more generally to every kind of edutainment applications, either in theme parks on in general public home entertainment (video games).

The research institute SINTEF and the SME CompuTouch from Norway have developed prototype software for a specific use of CompuTouch's tactile motors, thus combining the expertise of CompuTouch in tactile motors and the expertise of SINTEF in 3D geometry. This software allows it to interrogate digital 3D objects and to experience the feedback through the sense of touch using the tactile motors. The 3D objects are given either as triangulations (a large set of triangles) or in the standard CAD (computer-aided design) format of B-spline patches, but other geometric representations could easily be included. Computable properties of these 3D objects, such as curvature or smoothness, can be translated into tactile feedback patterns defined by the user. This is not only the case for the geometry, i.e. the spatial description, of an object, but includes the texture and its properties as well. One significant and novel aspect of the work is the use of a PC's graphics hardware as a computational resource to speed up the necessary computations enormously. This prototype software therefore opens up large possibilities for use in virtual reality applications to feel objects and their properties through the fingertips, in addition to seeing them. This approach will offer significant gains in perception possibilities, for example for medical environments, the presentation of cultural heritage objects in museums (which was the test application for the MUVII project) and the computer games industry. A special focus will be put on the needs of the vision-impaired and blind, since this technology offers unique chances for them to relate to digital objects by getting haptic (touch) feedback. Potential partners are invited to work together with SINTEF and CompuTouch to adapt this software to a specific application scenario, for example in the areas indicated above, but not limited to them, and bring this technology to commercial use.

Thanks to the CEA LIST, extensive knowledge of haptics, specific robotics design methodology, and dimensioning and optimisation tools, a major step was covered towards the development of intuitive and effective wearable haptic devices, with a variety of new potential commercial applications. In particular, an innovative and patent pending 6DOF wearable haptic interface with force and tactile feedback was developed. It allows fine and intuitive finger manipulation of virtual objects. Wireless 6DOF tracking allows free hand movements (range of motion exceeds fingers one while the device can be freely moved over more than 1 meter). High-end actuation stage enables around 5N high quality force feedback, allowing realistic interactions with the virtual environment. Moreover, tactile actuators also allow shape and textures exploration. These breakthroughs allow better performances than State of the Art research and commercial products. To date, it was used in an Interactive Kiosk Demonstrator with educational applications. More generally, it could be applied to every kind of edutainment applications (theme parks or home entertainment). It could also be associated with a force feedback master arm or an arm exoskeleton to allow richer interactions, with potential applications in industry (virtual prototyping and assembly, ergonomics and safety standards definition) or in re-education.

Tactile technology, what is it? Walking in the park with your girlfriend and you squeeze her hand, and she squeezes back that is tactile feedback. At Laval Virtual 2004 many users expressed surprise and enthusiasm over the added tactile information provided by the MUVII Interactive Theatre and Kiosk Devices. This new communication channel opens up five basic areas for exploitation. Using the Tactile sense where it surpasses the visual: This is a special market area. Think of a large polished surface, with just a few corns of sand stuck to it. Even in broad daylight it is very difficult to assess if there are such imperfections, and if so how many. However by just passing ones fingers lightly over the surface we immediately "feel" a each single sand corn. Only then due we "zoom in" to look at it. Finding the "imperfections" is something our sense of touch is very good at, so we should use it. Application areas include, meteorological data, geological data and also nanosurfaces to find one atom in a million that sticks up. We have constructed and patented a small device that converts electrical signals to small movements that can be felt. This device is referred to as a tactile motor; it has a small active surface about the size of a fingertip. Several tactile motors can be mounted in a tactile pointing device to stimulate the operator's fingertips. In this way the fingers may simultaneously "feel" the metrological or geological data or the special atoms on a nanosurface, complementing the visual. Once located the normal visual techniques can be used for further study.

The IKD Platform is providing the IKD Development Framework and the IKD Run-time Platform. The architecture of the IKD Platform is generally very heterogeneous and implies hard and long effort to combine and integrate data corresponding to each modality (visual, sound, force, tactile). Several modules or "engines" comprise the IKD Platform: Application & Graphics Engine (based on Virtools), Physics Engine (Vortex), Tracking System, 3D Audio Engine, force-feedback controllers and tactile-feedback controllers. The multi-modality of haptics, 3D-audio and 3D-graphics, all combined together, provide a unique and innovative platform for the rapid development and efficient running of advanced 3D haptic applications. The heart of the Development Framework is an efficient commercial authoring tool (Virtools) in which we have integrated some innovative MUVII haptic specific features like haptic rendering, haptic sounds, etc. During development stage, IKD development framework offers a full-centralized management for each kind of supported media in order to simplify the application development process. This approach allows the discarding of most of redundancies and insures data consistency all along the system. Therefore, the application developer can focus on the application development without being disturbed by a lot of technical issues. Moreover, MUVII IKD platform provides rapid application development, through a library of predefined generic objects & behaviours.

IKA-SIM is a real-time auditory virtual environment generator. It's software to aural auditory scenes in virtual rooms. All kind of rooms or environments can be predefined. The listener himself can move in real-time in this environment while listening to several sound sources, which also can be moved interactively. The listener perceive a fully three-dimensional sound reproduction including real-time updated early reflections and late-reverb. The possible output formats are headphones and an array of loudspeakers. This software was, in MUVII, fully integrated in a multimodal environment together with the haptic and visual modality. There is a general application interface (based on RPC) to access IKA-SIM over Ethernet.

Two prototype Educational/Entertainment 3D applications for schoolchildren of varying ages - focusing on the fields of physics and mechanics (gears), designed, developed and evaluated by the University of Patras especially for the multi-sensory environment of the MUVII IKD. The one application is about "Newtonian Physics, trajectories & the Solar System" were the user navigates through the solar system, collects information about it and interacts with the various elements of which it consists such as the planets, satellites, comets and asteroids. The user experiences the effect of the forces when accelerating objects as well as the strength of the gravitational forces applied to objects at different distances. The second application is about "Model Assembly & Gears" were users learn about toothed wheels, gears and their applications and they can exercise with the assembly of a clock and a watermill. To enhance the haptic experience of IKD applications, special visual, audio and haptic interaction metaphors have been specified in order to give emphasis on events that provide users with haptic and tactile feedback. These metaphors facilitate the users to better immerse in the virtual world and perceive the concepts of the Educational Applications. The IKD applications were used and verified by more than 200 school children (aged between 10-17 years old) in Patras and demonstrated to users of various ages, (kids, teenagers and adults) at Laval Virtual in May 2004. Despite the fact that the device was aimed at children of ages above 10 years old, it occurred that even younger children (e.g. 7-10 years old) could use it effectively. The trials were based on questionnaires and the results are analysed in a special report and show that IKD could offer great improvements to the existing teaching methods, thus enhancing the quality of educational procedures. All teachers and esp. pupils were very enthusiastic and attracted by the IKD and its applications. The applications helped users to concentrate on their aim. Currently there are no applications similar to the ones we have specified and implemented in the framework of the MUVII project, incorporating characteristics like independent force-feedback on two fingers, the combination of force-feedback and tactile feedback, efficient 3D-sound sub-system, targeting children of varied school age, efficient haptic interaction metaphors etc.

The initial goal of MUVII project was to prove the possibility to design two kinds of devices using haptic metaphors at a low price. The project has reached this goal with the creation of ITD and IKD devices. We have proved with the duplication workpackage that the cost of this device is low as far as the price of the duplicated prototype is lower than industrialized products already present on the international market. To go further, it was necessary to set up a new start-up company to industrialise and commercialise the new products. An alternative solution was also to transfer the technology to an existing company with the condition for this company to be installed in the Laval Science Park. We have convinced the most relevant company, which is Haption, a spin-off of CEA that has developed the Virtuose haptic arm to establish their offices in Laval and to begin a new activity based on haptic metaphor Technology. This activity is now going to start and we hope to have a commercial product within the next 2 years. The first product will be based on IKD prototype which can be interfaced with other existing products and that will also be installed in the SAS Cube environment operated by CLARTE (Centre Lavallois de Ressources Technologiques). This is why we consider that this result is very important and confirm the complete success of MUVII project.

Tactile technology, what is it? Walking in the park with your girlfriend and you squeeze her hand, and she squeezes back that is tactile feedback. At Laval Virtual 2004 many users expressed surprise and enthusiasm over the added tactile information provided by the MUVII Interactive Theatre and Kiosk Devices. This new communication channel opens up five basic areas for exploitation. 1) Helping handicapped and visually impaired PC users: Imagine a tactile pointing device (mouse) where the users physically "feel" the edges of a window as it is crossed. Or a distinct tactile motion pattern for the "hour glass cursor" and other "buttons" as they navigate over the screen. Ordinary users and those suffering from RSI would "feel" this cursor shape changes and use a reflex action to push the button. We have constructed and patented a small device that converts electrical signals to small movements that can be felt. This device is referred to as a tactile motor; it has a small active surface about the size of a fingertip. In a tactile mouse one or more tactile motors are mounted in the buttons where finger tip(s) normally are placed. In this way the fingers simultaneously "feel" the screen information as the cursor is moved over the screen. We have a working software application that connects the operating system cursor events to a set of user configurable tactile motion patterns. Thus tactile communication can be instantly available without any modification to existing or new software.

Tactile technology, what is it? Walking in the park with your girlfriend and you squeeze her hand, and she squeezes back that is tactile feedback. At Laval Virtual 2004 many users expressed surprise and enthusiasm over the added tactile information provided by the MUVII Interactive Theatre and Kiosk Devices. This new communication channel opens up five basic areas for exploitation. Using "Touch" as well as sight to operate equipment: When operating or steering equipment from cars to planes to cranes, the operators visual attention should normally be outside the equipment. Tactile technology expands the operator interface to receive additional information through a separate touch channel. This is especially relevant in critical situations where important information may otherwise be "overlooked". For example the car driver could receive navigation information through his fingertips instead of needing to look at a screen. The crane driver could even "feel" to pick up or place an object where his vision was obscured. A ROV operator might "feel" where he cannot see. We have constructed and patented a small device that converts electrical signals to small movements that can be felt. This device is referred to as a tactile motor; it has a small active surface about the size of a fingertip. One or more tactile motors can be mounted in steering wheels, joysticks or manipulating handles under the operator's fingertips. In this way the fingers simultaneously "feel" the sensor and/or navigation information while his vision remains 100% outside of the equipment.

Tactile technology, what is it? Walking in the park with your girlfriend and you squeeze her hand, and she squeezes back that is tactile feedback. At Laval Virtual 2004 many users expressed surprise and enthusiasm over the added tactile information provided by the MUVII Interactive Theatre and Kiosk Devices. This new communication channel opens up five basic areas for exploitation. Adding tactile feedback to PC games and Internet surfing: This is a huge mass-market area. Think of a new tactile pointing device, which not only enables you to position the cursor (mouse) but also allows the application, and the operating system to give tactile feedback to the user through his or her finger tips. Force feedback is already part of the gamer's world. To this we can add the subtlety of tactile feedback for surface contours and textures, or tactile motion patterns that can be directly associated with different game states or occurrences. Similarly for Internet sales, tactile technology could be used to let users, feel textures of materials and clothes etc. We have constructed and patented a small device that converts electrical signals to small movements that can be felt. This device is referred to as a tactile motor; it has a small active surface about the size of a fingertip. Several tactile motors can be mounted in a tactile pointing device to stimulate the operator�s fingertipsh. In this way the fingers simultaneously �feel� game or texture etc. information complementing the visual.

The project lead to the development of user interfaces providing haptic feedback to applications aimed at large public, and potentially large throughput. More specifically, the area of edutainment has been assessed, opening new man-machine interfaces to large public. The goal is to add new feedback to virtual reality applications, to improve the existing means of feedback (visual, aural), to reach the maximal immersion of the visitors in the virtual worlds they are discovering. It is the very first time, in the world, that such a development is achieved, as accurate haptic feedback was until this project, restricted to the fields of science and industry, i.e. research. The specifications of the project imply that the unique features of providing those feedback to large, untrained public. The project is aimed at the market of "edutainment", the bleeding area between museums (sciences museums in particular), and theme parks. The project led to an integrated demonstrator of the technology, involving cultural assets dissemination. It consists in a paleontological search simulator where the public discover Russian vestiges of the Past. The visitor uses the ITD device to dig the soil, to clean the virtual assets, and to ask question about the history of those assets; the presentation is followed as a group, and involves group interaction.

The MUVII IKD (Interactive Kiosk Demonstrator) is one out of the two major results of IST MUVII Project. This result integrates several other results of this project. It consists of three main parts: 1) The IKD Haptic Device, an innovative and patent pending 6DOF wearable haptic interface with force and tactile feedback that allows fine and intuitive finger manipulation of virtual objects; 2) The IKD Development and Run-Time Platform a unique and innovative platform for the rapid development and efficient running of advanced 3D haptic applications; 3) Two prototype demonstrator educational/entertainment applications for schoolchildren, namely "Newtonian physics, trajectories and the Solar System" and "Virtual Model Assembly & Gears". The user of IKD by using the IKD Haptic Device can investigate and explore various 3D objects and feel their material, surface, size, shape, etc. or select, pick-up, hold, move, orient and release/place objects and feel forces on her fingers (weight, torque, collisions, etc.). Several modules or "engines" comprise the IKD Platform: Application & Graphics Engine (based on Virtools), Physics Engine (Vortex), Tracking System, 3D Audio Engine, force-feedback controllers and tactile-feedback controllers. Our approach to use an Interactive Haptic-based Kiosk for educating children is not very common. The opportunity of having a natural "look and feel" environment for teaching purposes is very promising indeed. MUVII IKD have demonstrated new interaction paradigms in a novel integration of interaction modalities: 3D-vision, 3D-audio and haptic (force and tactile) feedback. The IKD applications and the whole IKD have been tested and verified by more than 200 school children (aged between 10-17 years old) in Patras and demonstrated to users of various ages, (kids, teenagers and adults) at Laval Virtual in May 2004. The trials showed that IKD could offer great improvement to the existing teaching methods. All teachers and esp. pupils were very enthusiastic and attracted by the IKD and its applications. The applications helped users to be concentrated on their aim. Currently there are not applications similar to the ones we have specified and implemented in the framework of the MUVII project, incorporating characteristics like independent force-feedback on two fingers, combination of force-feedback and tactile feedback, efficient 3D-sound sub-system, targeting children of varied school age, efficient haptic interaction metaphors etc.

Overview of MUVII ITD. From the various tests we carried out (some sessions were the topic of an important marketing study we can summarize that the added value of the haptic device in interactive theatres comes from: - More intuitive access to the content: to grasp an object, the user has simply to carry on the action of grasping. The action on a joystick is most of the time a mediator to another action; on the contrary, the ITD gives direct access to the action. As a consequence, there is also lower need of training, and therefore an immediate enjoyment/satisfaction in the experience. - More freedom of use: the device is not jointly attached to a console as the joystick or a button is; the new ITD gives therefore more autonomy to the user, who is not stick to the hardware ergonomics. It the case of the ITD, this is the reverse: the ergonomics follows the movements of the user; the user can displace his/her arms and hands in his normal range of operation. - More immersion in the interactive show: the better intuitiveness leads to a truly physical involvement of the visitors, the involvement into the content goes beyond other interfaces. Also, stimuli of the tactile and force adds new components to the multi-sensorial experience proposed to the visitors; it contributes to the sensorial captivation of the visitors, that are really surrounded by the experience: image, sound, touch, forces. Most of the senses are now captivated by the interactive experience. The experience leaves a stronger impact and memories to the visitor. - More innovative approach: the innovative character of the device is itself a factor of added value in an environment in demand for new, interactive experiences; in particular, tactile and force feedback are until now not the most offered to visitors. Overview of MUVII IKD. The opportunity of having a natural "look and feel" environment for teaching purposes is very promising indeed. MUVII IKD demonstrates new interaction paradigms in a novel integration of interaction modalities: 3D-vision, 3D-audio and haptic (force and tactile) feedback. The most current educational applications are based on audio and visual metaphors. IKD applications and the corresponding trial results proved the concept that haptics metaphor - that consists of force and tactile feedback - facilitate the users to better immerse in the virtual world and perceive the concepts of the Educational Applications. From a constructivist point of view what is really needed in several educational sectors, like physics, is a chance for the students to experience the effect of simple mechanics, and thereby allow them to gradually (but steadily) alter their preconceived ideas to the correct ones in a way that these will remain steady over time. MUVII IKD Virtual-experimental set-up allows us for the first time to give to the students a chance to feel such an experience. The H3DI allows the user to feel the gravitational forces, the force-feedback of throwing objects and the tactile-feedback of the various surfaces.

This result is about what users, esp. people in the education sector like pupils and teachers, expect and/or require from a haptic device and its accompanied applications. It is the outcome of the gathering, analysing and structuring of the user requirements for haptic devices and haptic educational applications. In Education the sense of force-feedback and tactile-feedback can offer great improvements to the existing teaching methods, thus enhancing the quality of education procedures. Developing Haptics Educational Applications is an under development research field and very few non-commercial applications have been developed so far. In our research and study, the applications and the device are oriented to each other and both to educational purposes. More specifically this result includes two main parts: 1) Specifications for the haptic device: basic characteristics of the device, user interaction (input/output) with the device, and device constraints esp. regarding the varied ages of the final users (children as well as adults). 2) Functional analysis of haptic applications: some preliminary functionality for the IKD applications (like view point management, user navigation, objects manipulation by users or with tools, etc.), and for those actions that affect the H3DI design and functionality we provide the expected Input (H3DI movement and press of buttons) and Output (force and tactile feedback). This analysis stressed the importance of grasping, manipulating and throwing objects in the virtual world. Moreover, due to the educational nature of IKD applications, realism is very important. We focused therefore on fine manipulation with fingers.

Tactile technology, what is it? Walking in the park with your girlfriend and you squeeze her hand, and she squeezes back that is tactile feedback. At Laval Virtual 2004 many users expressed surprise and enthusiasm over the added tactile information provided by the MUVII Interactive Theatre and Kiosk Devices. This new communication channel opens up five basic areas for exploitation. 2) Helping surgeons to "feel" as they operate using robotic equipment: In open surgery, the surgeons are able to feel the pulsing of a nearby artery as his scalpel approaches it, with tactile feedback we can recreate such a lost sense. Tactile feedback can be used in a simulated surgery training environment (see LOI from SimSurgery), and by adding small pressure/force sensors to the laparoscopy operating equipment we can give the surgeon tactile feedback. In a logical extension tele-surgery would be the next step. We have constructed and patented a small device that converts electrical signals to small movements that can be felt. This device is referred to as a tactile motor; it has a small active surface about the size of a fingertip. One or more tactile motors are mounted in the laparoscopic control handles where fingertips normally are placed. In this way the fingers simultaneously "feel" the sensor information as the surgical instrument is moved. Information from the sensors or the simulation model is converted to tactile movements so the surgeon can feel contours, textures and even the pulsing of an artery.

The project lead to the development of user interfaces providing haptic feedback to applications aimed at large public, and potentially large throughput. More specifically, the area of edutainment has been assessed, opening new man-machine interfaces to large public. The goal is to add new feedback to virtual reality applications, to improve the existing means of feedback (visual, aural), to reach the maximal immersion of the visitors in the virtual worlds they are discovering. It is the very first time, in the world, that such a development is achieved, as accurate haptic feedback was until this project, restricted to the fields of science and industry, i.e. research. The specifications of the project imply the unique features of providing that feedback to a large and untrained public. The project is aimed at the market of edutainment, the museums (sciences museums in particular), and theme parks. The project led to an integrated demonstrator of the technology, involving cultural assets dissemination. It consists in a paleontological search simulator where the public discover Russian vestiges of the Past. The visitor uses the ITD device to dig the soil, to clean the virtual assets, and to ask question about the history of those assets; the presentation is followed as a group, and involves group interaction.

Exploitable results

Share this page

Download