Skip to main content

SMARTSKETCHES: A multimodal approach to improve usability in the early stages of product design


This result delivers a framework integrating the different components developed in the Smart Sketches project. The framework consists out of: - A simple generic extensible interface specification - Templates for creating services that potentiate collaboration through the use of industry standard web services - A set of methodologies compliant to current trends in the industry - Enabling seamless integration and collaboration strategies between heterogeneous environments and applications. The current status of the framework is for internal use at the moment being extensively tested and developed upon by project partners. In the future, we expect to deliver an enhanced version result of partner contributions. This will enable future R&D projects to easily deploy distributed/collaborative solutions.
This result delivers a set of techniques integrated in a prototype currently used to open a new commercial area on Mind’s CAD/CAM portfolio. The integration work between the prototype and components developed by partners was done using a framework that is also a result of the SmartSketches project. Sketching techniques extensively tested during the project extent were aimed at a different user target, currently forgotten by the CAD/CAM industry. We are referring to designers/stylists or anyone without specific computer graphics expertise but, nevertheless, professionals working in the real industrial world with creative tasks. By exposing such user target to those techniques we expect to overcome current common belief that using CAD/CAM packages will in, some way, restrict creativity and artistic conception. The specific feedback of this result with industrial partners will give approach vectors to other industrial markets.
- The SketchAR system (as a whole): SketchAR is one of the first sketching and styling tools with allows working not only in Virtual Reality (VR) but also in Augmented Reality (AR). As such we see some market potential. During the SmartSketches project we carried out a market analysis and worked out a business plan together with the EBS (European Business School). Presentations and demonstrations have been performed to venture capitalists. At the moment the conditions to find venture capital are negatively influenced by the economic situation. In upcoming projects we will further develop the SketchAR system and continue to follow our spin-off ideas. - Handed tape drawing: Within SmartSketches the tape drawing approach commonly used in car styling has been transposed into a 2-handed virtual tape drawing approach with allows to create smooth curves on planes but also in 3D space. Recently a finger tracking approach has been implemented which allows together with speech commands to do virtual tape drawing on a PowerWall. This approach has proven useful in the user tests. Thus, a stand-alone system/module can be extracted from SketchAR with dedicated/limited functionality just providing 2-handed virtual tape drawing and some file I/O functionality. ItalDesign has already shown interest in such a module/system. - Curve ans surface sketching and over-sketching functionality During the SmartSketches project a couple of easy-to-use curve and surface sketching and over-sketching algorithms have been developed not common in todays styling tools, e.g. stroke analysis and automatic spitting, curve over-sketching, etc. Those modules/funcionalities can be further exploited in the future. Discussions with ICEM have been started and go on. - Multi-modal input SketchAR provides different input methaphors, namely: -- Personal interaction panel (2D menus embedded in 3D based on Studierstube functionality) -- Ring menus (3D menus that pop up upon request) -- 3D gestures (the 2D gesture recognizer has been extended to do 3D gesture recognition) -- Speech input and output Ring menus, gestures and speech I/O can be individually used by other components or can be bundled for re-use. This could be valuable for other parties to not re-implement base functionality by there own. - Integration back-bone SketchNet Based on the integration framework (Web Services, back-end data base) an integration back-bone has been implemented which provides functionality to couple multiple sketching applications for collaborative work. SketchNet consists of the SketchNet API for storing and retrieving shapes in/from the back-bone DB and distributing events whenever the DB is accessed. Sketching applications that comply with the SketchNet API can easily be facilitated to exchange information via Web Services, e.g. for online collaboration. The application just has to access the Web Service when creating shapes and has to register for events when other applications have created or modified a shape in the DB. For new types of shapes the DB structure has to be updated. - Cooperative SketchAR The combination of SketchAR and SketchNet allows for a collaborative use of SketchAR. Spatially dispersed users can work together with SketchAR. In the early stages of SmartSketches FIAT raised the requirement for online collaboration in sketching and styling applications. There was no workpackage / effort foreseen to follow-up on this requirement. As a side effect the integration framework finally allows for (a simple form) of online collaboration. Since there seam to be a need for collaborative styling in the industry we see a good chance for further exploitation.
The project result is an installable prototype version of ICEM Style, which can be connected to input devices and software device drivers developed by IGD and Barski Design. The integration of SmartSketches components to ICEM software (called ICEM Style+) allows the support of input devices working in immersive environments like 3D pen allowing designers to work in a virtual 3D space on a CAD model ("Cyberstilo"), immersive modelling techniques to move design work into virtual environments which come as close as possible to the non-virtual “real” working techniques, but with all the advantages of the computer and software plugins allowing CAD packages to extract usable geometry representations from ambiguous strokes on tablets, Powerwalls and normal desktop computer screens. One special result is a communication protocol, which allows it to combine standard CAD applications with immersive visualisation and editing techniques. The protocol is open and can easily be implemented by software vendors as well as by vendors of immersive hardware components. During the project IGD and ICEM have developed together not only a feasibility study, also a demonstration toolkit to the potential end users inside the project and based on this a demonstration toolkit for potential end users of the software outside the project.
CAD systems have yet to become usable at the early stages of product ideation, where precise shape definitions and sometimes even design intentions are not fully developed. To overcome these limitations, new approaches, which were called Calligraphic Interfaces, use sketching as the main organizing paradigm. Such applications rely on continuous input modalities rather than on discrete interactions characteristic of WIMP interfaces. However, replacing direct manipulation by sketching alone poses very interesting challenges. While the temptation to follow the paper-and-pencil metaphor is great, free-hand sketch recognition remains an elusive goal. Further, using gestures to enter commands and sketches to draw shapes requires users to learn a command set - sketches do not enjoy the self-disclosing characteristics of menus. Moreover, the imprecise nature of interactions presents additional problems that are difficult to address using present-day techniques. We approach the three problems outlined above through a combination of different paradigms: First, a calligraphic sketching metaphor provides for a paper-like interaction. Second, dynamic menus - expectation lists - handle ambiguity and try to expose the state of the application without interfering with the task. Third, an incremental drawing paradigm allows precise drawings to be progressively constructed from sketches through simple constraint satisfaction. Finally, reducing instruction set and command usage allow for a simple and learnable approach in contrast with the complexity of present-day interactive systems. Our prototype - GIDeS - embodies these approaches and usability testing carried so far yielded very encouraging results.
Retrieving drawings using sketches comprises three main tasks: The first is automatic indexation of technical drawing databases. To achieve this we have developed techniques to simplify drawings based on their geometric features. The second task, which constitutes the central component of the information retrieval system, yields methods to match hand-drawn sketches to technical drawings present in the database and associate a measure of similarity to retrieved objects, based on a common symbolic description. The third task has developed a user interface to query drawing databases and CAD libraries, using hand-drawn sketches.
In the SmartSketches project modelling techniques are developed suitable for creating and modifying free-form surfaces in 3D space using commercially available virtual environment systems together with innovative use of calligraphic/immersive devices to meet designers' requirements. Barskidesign is responsible for the product design development of the 3 D tools. This includes the 3 D Pen called "Cyberstilo", a 3 D Mouse called "CyberStone", a PIP (an personal interactive panel) and a Tapefinger. In the 1st stage we developed a system of different 3 D tools, easy to understand and easy to handle, with a family character and from technical side with less different parts for various applications and easy production. The creative work starts with analysis, sketches and foam model making in scale 1:1. various user tests at Fraunhofer IGD were the requirement for the detailing design stage, construction and the rapid prototyping of the tools. The cyberstilo was the first working prototype with all functionalities, presented at the EUROMOLD in Frankfurt, December 2003 and at the CEBIT March 2004 in Hanover.