The KRISTINA Consortium carried out substantial work on all topics related to its goal to develop a multilingual conversation agent with social and cultural competence. A generic dialogue manager with the ability to fuse multimodal knowledge structures and to adapt to the cultural background of the user by dialogue strategies trained on culture-specific dialogues has been implemented. In the context of speech recognition, a total of 53 hours of manually transcribed data were used to adapt VR’s pre-existing speech-to-text models to real-time contexts. The adaptation led to accuracy gains between 24% and 50% relative, measured on KRISTINA data. Furthermore, the feature analysis, acoustic models and speech recognition have been tuned to reduce the latency (<1 second in 75% of the cases) for all KRISTINA languages. For syntactic analysis of transcriptions, a Stack LSTM-based model has been developed and substantial treebanks for German, Polish, Spanish and Turkish have been annotated. For semantic analysis, deep-syntactic dependency parses are projected onto FrameNet-based structures, which are then translated into OWL representations. The translation is aligned with the DOLCE Ultralight ontology and with the patterns of the Descriptions and Situations ontology. For non-verbal communication analysis, over 5 hours of dialogue recordings have been annotated in terms of Valence and Arousal (V-A) labels; to ensure reliable ground truth, a new consensus methodology has been developed. Several non-verbal modules have been realized: (i) frame-wise real-time V-A detection from audio signals with Neural Networks; (ii) emotional state derivation based on real time facial imaging, which generates output in the V-A space and in terms of prototypical facial expressions; (iii) hand and body posture tracking and heuristic derivation of the level of Arousal of the user based on the tracking outcome; (iv) body model for recognition of pointing gestures based on hand positions relative to the body. For fusion of the input from different modules, a novel asynchronous event-based methodology has been researched. For knowledge representation, ontological models have been defined and a knowledge base (KB) has been populated to capture background information, user models, conversation histories, etc. A flexible context extraction algorithm has been developed that employs semantic similarity metrics and graph expansion techniques to match user statements against KB structures. A conversation context aware reasoning module has been implemented that is instrumental in retrieving relevant information from the KB that is to be communicated to the user. To ensure the availability of required background information in which a user might be interested, dedicated search engines have been developed to retrieve information from the web (entire pages or passages) and from social media. In the area of multimodal communication generation, mode selection, language generation, and expressive avatar creation has been worked on. A cognitively motivated rule-and classified-based mode selection has been implemented that treats voice, mimics and gestures as equal (instead of assuming voice to be the dominant mode to which the other modi are added). In the context of language generation, this involved: multiple SVM-based syntactic sentence generation, extension of lexical and grammatical resources for multilingual rule-based graph transducer-based sentence generation, NN-based punctuation generation, and expressive prosody generation. For expressive avatar creation, a web-based pipeline for the generation of characters has been set up and a number of human-like characters were created. To interface with modality selection, a BML Realizer and a BML Planner have been developed.
