Multi-aspect and diffErenTiable Evaluation of Rankings

Informacje na temat projektu

METER

Identyfikator umowy o grant: 893667

DOI

10.3030/893667

Projekt został zamknięty

Data podpisania przez KE 6 Sierpnia 2020

Data rozpoczęcia 1 Października 2020

Data zakończenia 30 Września 2022

Finansowanie w ramach

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Koszt całkowity

€ 207 312,00

Wkład UE

€ 207 312,00

Koordynowany przez

KOBENHAVNS UNIVERSITET
Denmark

Periodic Reporting for period 1 - METER (Multi-aspect and diffErenTiable Evaluation of Rankings)

Okres sprawozdawczy: 2020-10-01 do 2022-09-30

Podsumowanie kontekstu i ogólnych celów projektu

Information Retrieval (IR) deals with the automatic retrieval and ranking of information conveying items, which are relevant to a specific information need. Search engines are the most popular and well-known examples of IR systems.
Current evaluation procedures for IR systems mainly focus on the relevance of an item (e.g. Web page) with respect to the user need, but this might not be enough. For example, credibility and correctness should be considered to avoid returning misinformation and potentially harmful content.
This project investigated how to evaluate the performance of IR systems beyond relevance. The first goal was to extend IR evaluation measures to deal with multiple aspects. The second goal was to thoroughly analyze the properties of measures able to account for multiple aspects so that such measures could be integrated in IR systems and lead to better performance.

Prace wykonane od początku projektu do końca okresu sprawozdawczego oraz najważniejsze dotychczasowe rezultaty

The first goal was to define IR evaluation measures able to account for multiple. We proposed a theoretical formal framework and defined a new family of evaluation measures able to consider multiple aspects simultaneously, e.g. relevance, credibility, correctness, etc. We experimentally evaluate such framework showing that it overcomes the pitfalls of previously proposed measures.
The second goal was to investigate the properties of multi-aspect IR measures, for example how the measure score increases or decreases with respect to the amount, quality, and location of information in a result list returned by an IR system. We analysed our proposed framework for multi-aspect evaluation and mathematically show that it satisfies some formal properties, e.g. the measure score increases when a greater amount of high-quality information is retrieved and when it is placed close to the beginning of the ranked list.
These results are reported in several publications and were presented at conferences and invited seminars.

Innowacyjność oraz oczekiwany potencjalny wpływ (w tym dotychczasowe znaczenie społeczno-gospodarcze i szersze implikacje społeczne projektu)

This project has the potential for making societal impact because: 1) our multi-aspect evaluation approach can be used to account for several aspects and improve the effectiveness of IR systems in different domains; 2) a deep understanding of the properties of our evaluation approach allows to integrate such approach in IR systems and design systems that are able to account for multiple aspects when returning a ranked list of items to users.
Moreover, this project enhanced the career perspectives of the research fellow, who got a tenure track position at the host university.