Multi-aspect and diffErenTiable Evaluation of Rankings

Informazioni relative al progetto

METER

ID dell’accordo di sovvenzione: 893667

DOI

10.3030/893667

Progetto chiuso

Data della firma CE 6 Agosto 2020

Data di avvio 1 Ottobre 2020

Data di completamento 30 Settembre 2022

Finanziato da

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Costo totale

€ 207 312,00

Contributo UE

€ 207 312,00

Coordinato da

KOBENHAVNS UNIVERSITET
Denmark

Periodic Reporting for period 1 - METER (Multi-aspect and diffErenTiable Evaluation of Rankings)

Periodo di rendicontazione: 2020-10-01 al 2022-09-30

Sintesi del contesto e degli obiettivi generali del progetto

Information Retrieval (IR) deals with the automatic retrieval and ranking of information conveying items, which are relevant to a specific information need. Search engines are the most popular and well-known examples of IR systems.
Current evaluation procedures for IR systems mainly focus on the relevance of an item (e.g. Web page) with respect to the user need, but this might not be enough. For example, credibility and correctness should be considered to avoid returning misinformation and potentially harmful content.
This project investigated how to evaluate the performance of IR systems beyond relevance. The first goal was to extend IR evaluation measures to deal with multiple aspects. The second goal was to thoroughly analyze the properties of measures able to account for multiple aspects so that such measures could be integrated in IR systems and lead to better performance.

Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti

The first goal was to define IR evaluation measures able to account for multiple. We proposed a theoretical formal framework and defined a new family of evaluation measures able to consider multiple aspects simultaneously, e.g. relevance, credibility, correctness, etc. We experimentally evaluate such framework showing that it overcomes the pitfalls of previously proposed measures.
The second goal was to investigate the properties of multi-aspect IR measures, for example how the measure score increases or decreases with respect to the amount, quality, and location of information in a result list returned by an IR system. We analysed our proposed framework for multi-aspect evaluation and mathematically show that it satisfies some formal properties, e.g. the measure score increases when a greater amount of high-quality information is retrieved and when it is placed close to the beginning of the ranked list.
These results are reported in several publications and were presented at conferences and invited seminars.

Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora)

This project has the potential for making societal impact because: 1) our multi-aspect evaluation approach can be used to account for several aspects and improve the effectiveness of IR systems in different domains; 2) a deep understanding of the properties of our evaluation approach allows to integrate such approach in IR systems and design systems that are able to account for multiple aspects when returning a ranked list of items to users.
Moreover, this project enhanced the career perspectives of the research fellow, who got a tenure track position at the host university.