Multi-aspect and diffErenTiable Evaluation of Rankings

Información del proyecto

METER

Identificador del acuerdo de subvención: 893667

DOI

10.3030/893667

Proyecto cerrado

Fecha de la firma de la CE 6 Agosto 2020

Fecha de inicio 1 Octubre 2020

Fecha de finalización 30 Septiembre 2022

Financiado con arreglo a

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Coste total

€ 207 312,00

Aportación de la UE

€ 207 312,00

Coordinado por

KOBENHAVNS UNIVERSITET
Denmark

Periodic Reporting for period 1 - METER (Multi-aspect and diffErenTiable Evaluation of Rankings)

Período documentado: 2020-10-01 hasta 2022-09-30

Resumen del contexto y de los objetivos generales del proyecto

Information Retrieval (IR) deals with the automatic retrieval and ranking of information conveying items, which are relevant to a specific information need. Search engines are the most popular and well-known examples of IR systems.
Current evaluation procedures for IR systems mainly focus on the relevance of an item (e.g. Web page) with respect to the user need, but this might not be enough. For example, credibility and correctness should be considered to avoid returning misinformation and potentially harmful content.
This project investigated how to evaluate the performance of IR systems beyond relevance. The first goal was to extend IR evaluation measures to deal with multiple aspects. The second goal was to thoroughly analyze the properties of measures able to account for multiple aspects so that such measures could be integrated in IR systems and lead to better performance.

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

The first goal was to define IR evaluation measures able to account for multiple. We proposed a theoretical formal framework and defined a new family of evaluation measures able to consider multiple aspects simultaneously, e.g. relevance, credibility, correctness, etc. We experimentally evaluate such framework showing that it overcomes the pitfalls of previously proposed measures.
The second goal was to investigate the properties of multi-aspect IR measures, for example how the measure score increases or decreases with respect to the amount, quality, and location of information in a result list returned by an IR system. We analysed our proposed framework for multi-aspect evaluation and mathematically show that it satisfies some formal properties, e.g. the measure score increases when a greater amount of high-quality information is retrieved and when it is placed close to the beginning of the ranked list.
These results are reported in several publications and were presented at conferences and invited seminars.

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

This project has the potential for making societal impact because: 1) our multi-aspect evaluation approach can be used to account for several aspects and improve the effectiveness of IR systems in different domains; 2) a deep understanding of the properties of our evaluation approach allows to integrate such approach in IR systems and design systems that are able to account for multiple aspects when returning a ranked list of items to users.
Moreover, this project enhanced the career perspectives of the research fellow, who got a tenure track position at the host university.