Improving social competences of virtual agents through artificial consciousness based on the Attention Schema Theory

Informacje na temat projektu

ASTOUND

Identyfikator umowy o grant: 101071191

DOI

10.3030/101071191

Projekt został zamknięty

Data podpisania przez KE 14 Grudnia 2022

Data rozpoczęcia 1 Grudnia 2022

Data zakończenia 30 Listopada 2025

Finansowanie w ramach

• The European Innovation Council (EIC)

Koszt całkowity

€ 3 330 897,50

Wkład UE

€ 3 330 897,50

Koordynowany przez

UNIVERSIDAD POLITECNICA DE MADRID
Spain

CORDIS oferuje możliwość skorzystania z odnośników do publicznie dostępnych publikacji i rezultatów projektów realizowanych w ramach programów ramowych HORYZONT.

Odnośniki do rezultatów i publikacji związanych z poszczególnymi projektami 7PR, a także odnośniki do niektórych konkretnych kategorii wyników, takich jak zbiory danych i oprogramowanie, są dynamicznie pobierane z systemu OpenAIRE .

Rezultaty

Demonstrators, pilots, prototypes (3)

Architecture validated
"""Graziano/Turing test"" developed"

“Graziano/Turing test” developed

Cognitive plausibility and role of attention schema established

Documents, reports (6)

Publications describing novel conversational agent
Guidelines for an ethical roadmap for oversight of AI projects
Process of database curation established
Publications describing the conscious conversational agent
Website and project logo

Project logo and website ready

Publications describing the AST architecture

Websites, patent fillings, videos etc. (3)

SME vehicle for IP exploitation incorporated
Patents protecting the architecture and the conscious agent update

Patentability of the architecture and the conscious agent assessed and more patent filed

Study of patentability of the proposed architecture and conscious agent

Preliminary study on the patentability for protecting the architecture and the conscious agent

Data Management Plan (2)

RP2 Update of Data Management Plan

DMP updated

Data Management Plan

DMP generated

Other (3)

Proposal of core morality

Proposal of core morality and its assessment to ensure system safety and allow human oversight

Report on the impact of conscious artificial agents on society
Promoting equality and fairness through self-aware algorithms

Data sets, microdata, etc (2)

Datasets for training the conscious conversational agent
Datasets for training the conscious conversational agent update 2

Publikacje

Conference proceedings (35)

Context or Retrieval? Evaluating RAG Methods for Art and Museum QA System

Autorzy: Samuel Ramos-Varela, Jaime Bellver-Soler, Marcos Estecha-Garitagoitia, Luis Fernando D’Haro
Opublikowane w: Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology, 2025
Wydawca: Association for Computational Linguistics

Bias, Subjectivity and Norm in Large Language Models

Autorzy: Poibeau, T.
Opublikowane w: Aequitas (Fairness and Bias in AI), 2024
Wydawca: HAL

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark

Autorzy: Chen Zhang, Luis D’Haro, Chengguang Tang, Ke Shi, Guohua Tang, Haizhou Li
Opublikowane w: Findings of the Association for Computational Linguistics: EMNLP 2023, Numer 5579–5601, 2023
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2023.FINDINGS-EMNLP.371

Quelques observations sur la notion de biais dans les modèles de langue

Autorzy: Gallienne P., Poibeau T.
Opublikowane w: Actes de CORIA-TALN 2023. Actes de la 30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN), Numer Vol. 3, 2023
Wydawca: ATALA

"Awareness in robotics: An early perspective from the viewpoint of the EIC Pathfinder Challenge"" Awareness Inside''"

Autorzy: Cosimo Della Santina, Carlos Hernandez Corbato, Burak Sisman, Luis A. Leiva, Ioannis Arapakis, Michalis Vakalellis, Jean Vanderdonckt, Luis Fernando D’Haro, Guido Manzi, Cristina Becchio, A ̈ıda Elamrani, Mohsen Alirezaei, Ginevra Castellano, Dimos V. Dim
Opublikowane w: European Robotics Forum, 2024
Wydawca: European Robotics Forum
DOI: 10.48550/ARXIV.2402.09030

STEPI: System for Triplet Extraction of Personal Information in Dialogue Systems

Autorzy: Villa Monedero, M., Bellver, J., Ghebriout, M. I. E., De Cordoba, R., & D'Haro, L. F.
Opublikowane w: Iberspeech2024, 2024
Wydawca: ISCA

Multilingual Speech Emotion Recognition combining Audio and Text using Small Language Models

Autorzy: Jaime Bellver, Mario Rodríguez Cantelar, Marcos Estecha Garitagoitia, Ricardo De Córdoba Herralde, Luis Fernando D'Haro
Opublikowane w: Iberspeech2024, 2024
Wydawca: ISCA
DOI: 10.21437/IberSPEECH.2024-43

LLM-Driven Multimodal Fusion for Human Perception Analysis

Autorzy: Esteban-Romero, S., Martín-Fernández, I., Gil-Martín, M., Griol-Barres, D., Callejas-Carrión, Z., & Fernández-Martínez, F
Opublikowane w: MuSe'24: Proceedings of the 5th on Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Hum, 2024
Wydawca: ACM
DOI: 10.1145/3689062.36890

A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators

Autorzy: Zhang, Chen, Luis Fernando D'Haro, Yiming Chen, Malu Zhang, and Haizhou Li.
Opublikowane w: AAAI Conference on Artificial Intelligence, Numer vol. 38, no. 17, pp. 19515-19524., 2024, ISSN 2374-3468
Wydawca: Advancement of Artifcial Intelligence
DOI: 10.48550/arXiv.2312.15407

Improving Hand Pose Recognition using Localization and Zoom Normalizations over MediaPipe Landmarks

Autorzy: Remiro Pérez, Miguel Ángel, Gil Martín, Manuel and San Segundo Hernández, Rubén
Opublikowane w: 10th International Electronic Conference on Sensors and Applications session Sensors and Artificial Intelligence, 2023, ISSN 2673-4591
Wydawca: MDPI
DOI: 10.3390/ECSA-10-16215

DREAM: A Multicultural Multimodal Dataset Linking Dialogues and Realistic Image Sequences

Autorzy: Juan Mallo, Marcos Estecha-Garitagoitia, Ricardo Cordoba, Luis Fernando D'Haro
Opublikowane w: Proceedings of the Language Resources and Evaluation Conference, Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), 2026
Wydawca: European Language Resources Association (ELRA)
DOI: 10.63317/2V7B4XHS2D5G

Synthesizing Olfactory Understanding: Multimodal Language Models for Image–Text Smell Matching

Autorzy: Esteban-Romero, S., Martín-Fernández, I., Gil-Martín, M., & Fernández-Martínez, F.
Opublikowane w: Symmetry, Numer 17(8), 2025
Wydawca: MDPI
DOI: 10.3390/SYM17081349

Exploring Video Transformers and Automatic Segment Selection for Memorability Prediction

Autorzy: Martín-Fernández, I., Romero, S. E., Bellver-Soler, J., Gil-Martín, M., & Martínez, F. F.
Opublikowane w: MediaEval’23: Multimedia Evaluation Workshop,, 2024
Wydawca: CEUR

THAU-UPM at EmoSPeech-IberLEF2024: Efficient Adaptation of Mono-modal and Multi-modal Large Language Models for Automatic Speech Emotion Recognition

Autorzy: Esteban-Romero, S., Bellver-Soler, J., Martın-Fernández, I., Gil-Martın, M., D’Haro, L. F., & Fernández-Martınez, F.
Opublikowane w: Iberian Languages Evaluation Forum (IberLEF 2024) at SEPLN2024, 2024, ISSN 1613-0073
Wydawca: SEPLN

CVQA - Culturally-diverse Multilingual Visual Question Answering Benchmark

Autorzy: David Romero*👁️, Chenyang Lyu*👁️, Haryo Akbarianto Wibowo👁️, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco
Opublikowane w: NeurIPs, 2024
Wydawca: NeurIPs

Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models

Autorzy: Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H.
Opublikowane w: Findings of the Association for Computational Linguistics: ACL 2024, 2024
Wydawca: Association for Computational Linguistics
DOI: 10.18653/V1/2024.FINDINGS-ACL.80

Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12.

Autorzy: John Mendonça, J., Zhang, L., Mallidi, R., Lavie, A., Trancoso, I. D’Haro, L. F., Sedoc, J.
Opublikowane w: Proceedings SigDial 2025, 2025
Wydawca: SigDial

Beyond single-audio: Advancing multi-audio processing in audio large language models

Autorzy: Chen, Y., Yue, X., Gao, X., Zhang, C., D'Haro, L. F., Tan, R. T., & Li, H.
Opublikowane w: EMNLP2024, 2024
Wydawca: Association for Computational Linguistics

Video Memorability Prediction From Jointly-learnt Semantic and Visual Features

Autorzy: Martín-Fernández, Iván, Ricardo Kleinlein, Cristina Luna-Jiménez, Manuel Gil-Martín, and Fernando Fernández-Martínez
Opublikowane w: 20th International Conference on Content-based Multimedia Indexing, Orleans, France, 2023, ISBN 979-8-4007-0912-8
Wydawca: ACM

Interpreting Sign Language Recognition using Transformers and MediaPipe Landmarks

Autorzy: Cristina Luna-Jiménez, Manuel Gil-Martín, Ricardo Kleinlein, Rubén San-Segundo, Fernando Fernández-Martínez
Opublikowane w: 25th ACM International Conference on Multimodal Interaction, 2023, ISBN 979-8-4007-0055-2
Wydawca: Association for Machinery Computational
DOI: 10.1145/3577190.3614143

Overview of Robust and Multilingual Automatic Evaluation Metricsfor Open-Domain Dialogue Systems at DSTC 11 Track 4

Autorzy: Mario Rodríguez-Cantelar, Chen Zhang, Chengguang Tang, Ke Shi, Sarik Ghazarian, João Sedoc, Luis Fernando D’Haro, Alexander I. Rudnicky
Opublikowane w: Proceedings of The Eleventh Dialog System Technology Challenge, 2023
Wydawca: Association for Computational Linguistics

Orchestra: AI-Driven Microservices Architecture to Create Personalized Experiences

Autorzy: Bellver-Soler, J., Ramos-Varela, S., Guragain, A., Córdoba, R., D’Haro, L. F.
Opublikowane w: IWSDS 2026, 2026
Wydawca: ACL

Thaurus: An innovative multimodal chatbot based on the next generation of conversational ai. 

Autorzy: Estecha-Garitagoitia, M., Rodríguez-Cantelar, M., Ruiz, A. G., García, C. G. F., Romero, S. E., Conforto, C., ... & D’Haro, L. F.
Opublikowane w: Proceeding Amazon Alexa Socialbot Grand Challenge 5, 2023
Wydawca: Amazon

Can Large Sound Event Detection Models Be Accurately Adapted to Specific Acoustic Scenarios?.

Autorzy: Fernández-Castañón, R., Espinoza-Cuadros, F. M., Perero-Codosero, J. M., Sancho-Lozano, E., & Hernández-Gómez, L. A.
Opublikowane w: International Symposium on Distributed Computing and Artificial Intelligence, 2025
Wydawca: Springer Nature Switzerland
DOI: 10.1007/978-3-031-80946-0_4

Larger Encoders, Smaller Regressors: Exploring Label Dimensionality Reduction and Multimodal Large Language Models as Feature Extractors for Predicting Social Perception.

Autorzy: Martín-Fernández, I., Esteban-Romero, S., Bellver-Soler, J., Fernández-Martínez, F., & Gil-Martín, M.
Opublikowane w: Martín-Fernández, I., Esteban-Romero, S., Bellver-Soler, J., Fernández-Martínez, F., & Gil-Martín, M., 2024
Wydawca: ACM
DOI: 10.1145/3689062.368908

Frequency Analysis and Transfer Learning Across Different Body Sensor Locations in Parkinson’s Disease Detection Using Inertial Signals

Autorzy: Rey-Díaz, A., Martín-Fernández, I., San-Segundo, R., & Gil-Martín, M.
Opublikowane w: The 11th International Electronic Conference on Sensors and Applications, 2024
Wydawca: MDPI
DOI: 10.3390/ECSA-11-20507

A Comprehensive Analysis of Parkinson’s Disease Detection Through Inertial Signal Processing

Autorzy: Gil Martín, Manuel; Esteban Romero, Sergio; Fernández Martínez, Fernando; San Segundo Hernández, Rubén
Opublikowane w: reponame:Archivo Digital UPM, 2024
Wydawca: ICAART2024
DOI: 10.5220/0012360100003636

Design, Generation and Evaluation of a Synthetic Dialogue Dataset for Contextually Aware Chatbots in Art Museums

Autorzy: Rachidi, I., Ezzakri, A., Bellver-Soler, J., & D’Haro, L. F.
Opublikowane w: Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology, 2025
Wydawca: Association for Computational Linguistics

Cutting Through Overload: Efficient Token Dropping for Speech Emotion Recognition in Multimodal Large Language Models

Autorzy: Bellver-Soler, J., Rodríguez-Cantelar, M., Córdoba, R., & D’Haro, L. F.
Opublikowane w: Proceedings IWSDS 2025, 2025
Wydawca: Association for Computational Linguistics

Parkinson’s Disease Detection Through Inertial Signals and Posture Insights

Autorzy: Gil Martín, Manuel; Esteban Romero, Sergio; Fernández Martínez, Fernando; San Segundo Hernández, Rubén
Opublikowane w: Proceedings of the 16th International Conference on Agents and Artificial Intelligence | 16th International Conference on Agents and Artificial Intelligence - ICAART24 | 24/02/2024 | Rome, Italy, 2024
Wydawca: ICAART
DOI: 10.5220/0012451100003636

A Personalized, Multimodal AI Assistant for Enhancing Museum Visitor Experience

Autorzy: A. Guragain, J. Bellver-Soler, S. Ramos-Varela, D. Aragón-Diaz, Long, L., L.F. D’Haro
Opublikowane w: Proceedings ALA2025, 2025
Wydawca: CEUR

Multimodal Audio-Language Model for Speech Emotion Recognition

Autorzy: Bellver, J., Martín-Fernández, I., Bravo-Pacheco, J. M., Esteban, S., Fernández-Martínez, F., & D'Haro, L. F.
Opublikowane w: odyssey 2024 , 2024
Wydawca: ISCA
DOI: 10.21437/ODYSSEY.2024-41

ASSIST: A Multi-Agentic Framework for Human-Computer Interaction in Cultural Heritage Settings

Autorzy: Ramos-Varela, S., Guragain, A., Bellver-Soler, J., Aragón Díaz, D., Lin, L., & D’Haro, L. F.
Opublikowane w: International Conference on Information Management and Big Data, 2025
Wydawca: SIMBig 2025

Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models

Autorzy: Estecha-Garitagoitia, M., Zhang, C., Rodríguez-Cantelar, M., & D'Haro, L. F.
Opublikowane w: Proceedings IWSDS 2024, 2024
Wydawca: ELRA SigSDS

Multimodal and Multilingual Olfactory Matching based on Contrastive Learning

Autorzy: Esteban-Romero, S., Martín-Fernández, I., Bellver-Soler, J., Gil-Martín, M., & Fernández-Martínez, F.
Opublikowane w: MediaEval’23: Multimedia Evaluation Workshop, 2024
Wydawca: CEUR

Peer reviewed articles (11)

PoE: A Panel of Experts for Generalized Automatic Dialogue Assessment

Autorzy: Chen Zhang; Luis Fernando D'Haro; Qiquan Zhang; Thomas Friedrichs; Haizhou Li
Opublikowane w: IEEE/ACM Transactions on Audio, Speech, and Language Processing, Numer 31, 2023, ISSN 2329-9304
Wydawca: IEEE
DOI: 10.48550/arXiv.2212.08992

Sign Language Dataset for Automatic Motion Generation

Autorzy: Villa Monedero, María, Gil Martín, Manuel , Sáez Trigueros, Daniel, Pomirski, Andrzej and San Segundo Hernández, Rubén
Opublikowane w: Journal of Imaging, Numer 9 (12), 2023, ISSN 2313-433X
Wydawca: MDPI
DOI: 10.3390/JIMAGING9120262

Evaluating emotional and subjective responses in synthetic art-related dialogues: A multi-stage framework with large language models. 

Autorzy: Luna-Jiménez, C., Gil-Martín, M., D’Haro, L. F., Fernández-Martínez, F., & San-Segundo, R.
Opublikowane w: Expert Systems with Applications, Numer 255, 124524, 2024, ISSN 1873-6793
Wydawca: ELSEVIER
DOI: 10.1016/J.ESWA.2024.124524

Automatic Detection of Inconsistencies and Hierarchical Topic Classification for Open-Domain Chatbots

Autorzy: Mario Rodríguez-Cantelar, Marcos Estecha-Garitagoitia, Luis Fernando D’Haro,Fernando Matía, Ricardo Córdoba
Opublikowane w: Applied Sciences, Numer 13 (16), 2023, ISSN 2076-3417
Wydawca: MDPI
DOI: 10.3390/APP13169055

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Autorzy: Gunasekara, C., Kim, S., D'haro, L.F., Rastogi, A., Chen, Y.N., Eric, M., Hedayatnia, B., Gopalakrishnan, K., Liu, Y., Huang, C.W. and Hakkani-Tür, D.,
Opublikowane w: IEEE/ACM Transactions on Audio, Speech, and Language Processing, Numer 32, 4066 - 4076, 2024, ISSN 2329-9304
Wydawca: IEEE
DOI: 10.1109/TASLP.2024.3426331

Sign Language Motion Generation from Sign Characteristic

Autorzy: Gil Martín, Manuel , Villa Monedero, María, Pomirski, Andrzej, Sáez Trigueros, Daniel and San Segundo Hernández, Rubén
Opublikowane w: Sensors, Numer 23, 2023, ISSN 1424-8220
Wydawca: MDPI
DOI: 10.3390/S23239365

Dual Leap Motion Controller 2: A Robust Dataset for Multi-view Hand Pose Recognition

Autorzy: Manuel Gil-Martín, Marco Raoul Marini, Rubén San-Segundo, Luigi Cinque
Opublikowane w: Scientific Data, Numer 11, 2024, ISSN 2052-4463
Wydawca: Springer Science and Business Media LLC
DOI: 10.1038/S41597-024-03968-9

A dataset of synthetic art dialogues with ChatGPT

Autorzy: Gil-Martín, M., Luna-Jiménez, C., Esteban-Romero, S., Estecha-Garitagoitia, M., Fernández-Martínez, F., & D’Haro, L. F.
Opublikowane w: Scientific Data2052-4463, Numer 11(1),825, 2024, ISSN 2052-4463
Wydawca: Nature Portfolio
DOI: 10.1038/S41597-024-03661-X

Integrating Ethical Principles and Human Rights Based Approach in the EU Artificial Intelligence Act and the Council of Europe Convention on Artificial Intelligence: Interplay of Ethics and Law in the AI Regulation Debate

Autorzy: Irēna Barkāne, Celia Fernández Aller, Evelyne A. Tauchnitz, Iva Ramuš Cvetkovič, Manja Skočir, Aleš Završnik
Opublikowane w: Journal of the University of Latvia. Law, 2025, ISSN 2592-9364
Wydawca: University of Latvia
DOI: 10.22364/JULL.19.03

Unsupervised Mutual Learning of Discourse Parsing and Topic Segmentation in Dialogue

Autorzy: Jiahui Xu, Feng Jiang, Anningzhe Gao, Luis Fernando D'Haro, Haizhou Li
Opublikowane w: IEEE Transactions on Audio, Speech and Language Processing, Numer 33, 2025, ISSN 2998-4173
Wydawca: Institute of Electrical and Electronics Engineers (IEEE)
DOI: 10.1109/TASLPRO.2025.3623884

Testing Theory of Mind in GPT Models and Humans

Autorzy: James Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Alessandro Rufo, Guido Manzi, Michael Graziano, Cristina Becchio
Opublikowane w: Research Square, 2023, ISSN 2693-5015
Wydawca: Research Square
DOI: 10.21203/rs.3.rs-3262385/v1

Book chapters (2)

Aspectos éticos de la inteligencia artificial

Autorzy: : Tomás Bastarreche Bengoa, Alfonso Egea de Haro, María Mercedes Serrano Pérez, Celia Fernández Aller
Opublikowane w: Derecho y Justicia Global, Numer 3, 2025, ISBN 978-84-7991-660-2
Wydawca: Universitas

Ethical Risks and Opportunities in Designing Artificial Awareness

Autorzy: A. Tanevska et al. (including L.D'Haro, C. Fernandez, P. Maurizio, J. Salgado)
Wydawca: Springer

Other (2)

Attention Schema in Neural Agents

Autorzy: Liu, Dianbo; Bolotta, Samuele; Zhu, He; Bengio, Yoshua; Dumas, Guillaume
Opublikowane w: ArXiv, 2023
Wydawca: Cornell University
DOI: 10.48550/arxiv.2305.17375

“Living with IA - A Philosophical Toolkit for Navigating the Conceptual Challenges of Artificial Intelligence Systems”

Autorzy: Casati R., Maurizio P.
Wydawca: Institut Jean Nicod