Multi-aspect and diffErenTiable Evaluation of Rankings

Informations projet

METER

N° de convention de subvention: 893667

DOI

10.3030/893667

Projet clôturé

Date de signature de la CE 6 Août 2020

Date de début 1 Octobre 2020

Date de fin 30 Septembre 2022

Financé au titre de

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Coût total

€ 207 312,00

Contribution de l’UE

€ 207 312,00

Coordonné par

KOBENHAVNS UNIVERSITET
Denmark

Periodic Reporting for period 1 - METER (Multi-aspect and diffErenTiable Evaluation of Rankings)

Période du rapport: 2020-10-01 au 2022-09-30

Résumé du contexte et des objectifs généraux du projet

Information Retrieval (IR) deals with the automatic retrieval and ranking of information conveying items, which are relevant to a specific information need. Search engines are the most popular and well-known examples of IR systems.
Current evaluation procedures for IR systems mainly focus on the relevance of an item (e.g. Web page) with respect to the user need, but this might not be enough. For example, credibility and correctness should be considered to avoid returning misinformation and potentially harmful content.
This project investigated how to evaluate the performance of IR systems beyond relevance. The first goal was to extend IR evaluation measures to deal with multiple aspects. The second goal was to thoroughly analyze the properties of measures able to account for multiple aspects so that such measures could be integrated in IR systems and lead to better performance.

Travail effectué depuis le début du projet jusqu’à la fin de la période considérée dans le rapport et principaux résultats atteints jusqu’à présent

The first goal was to define IR evaluation measures able to account for multiple. We proposed a theoretical formal framework and defined a new family of evaluation measures able to consider multiple aspects simultaneously, e.g. relevance, credibility, correctness, etc. We experimentally evaluate such framework showing that it overcomes the pitfalls of previously proposed measures.
The second goal was to investigate the properties of multi-aspect IR measures, for example how the measure score increases or decreases with respect to the amount, quality, and location of information in a result list returned by an IR system. We analysed our proposed framework for multi-aspect evaluation and mathematically show that it satisfies some formal properties, e.g. the measure score increases when a greater amount of high-quality information is retrieved and when it is placed close to the beginning of the ranked list.
These results are reported in several publications and were presented at conferences and invited seminars.

Progrès au-delà de l’état des connaissances et impact potentiel prévu (y compris l’impact socio-économique et les conséquences sociétales plus larges du projet jusqu’à présent)

This project has the potential for making societal impact because: 1) our multi-aspect evaluation approach can be used to account for several aspects and improve the effectiveness of IR systems in different domains; 2) a deep understanding of the properties of our evaluation approach allows to integrate such approach in IR systems and design systems that are able to account for multiple aspects when returning a ranked list of items to users.
Moreover, this project enhanced the career perspectives of the research fellow, who got a tenure track position at the host university.