Multi-aspect and diffErenTiable Evaluation of Rankings

Projektinformationen

METER

ID Finanzhilfevereinbarung: 893667

DOI

10.3030/893667

Projekt abgeschlossen

EK-Unterschriftsdatum 6 August 2020

Startdatum 1 Oktober 2020

Enddatum 30 September 2022

Finanziert unter

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Gesamtkosten

€ 207 312,00

EU-Beitrag

€ 207 312,00

Koordiniert durch

KOBENHAVNS UNIVERSITET
Denmark

Periodic Reporting for period 1 - METER (Multi-aspect and diffErenTiable Evaluation of Rankings)

Berichtszeitraum: 2020-10-01 bis 2022-09-30

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Information Retrieval (IR) deals with the automatic retrieval and ranking of information conveying items, which are relevant to a specific information need. Search engines are the most popular and well-known examples of IR systems.
Current evaluation procedures for IR systems mainly focus on the relevance of an item (e.g. Web page) with respect to the user need, but this might not be enough. For example, credibility and correctness should be considered to avoid returning misinformation and potentially harmful content.
This project investigated how to evaluate the performance of IR systems beyond relevance. The first goal was to extend IR evaluation measures to deal with multiple aspects. The second goal was to thoroughly analyze the properties of measures able to account for multiple aspects so that such measures could be integrated in IR systems and lead to better performance.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

The first goal was to define IR evaluation measures able to account for multiple. We proposed a theoretical formal framework and defined a new family of evaluation measures able to consider multiple aspects simultaneously, e.g. relevance, credibility, correctness, etc. We experimentally evaluate such framework showing that it overcomes the pitfalls of previously proposed measures.
The second goal was to investigate the properties of multi-aspect IR measures, for example how the measure score increases or decreases with respect to the amount, quality, and location of information in a result list returned by an IR system. We analysed our proposed framework for multi-aspect evaluation and mathematically show that it satisfies some formal properties, e.g. the measure score increases when a greater amount of high-quality information is retrieved and when it is placed close to the beginning of the ranked list.
These results are reported in several publications and were presented at conferences and invited seminars.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

This project has the potential for making societal impact because: 1) our multi-aspect evaluation approach can be used to account for several aspects and improve the effectiveness of IR systems in different domains; 2) a deep understanding of the properties of our evaluation approach allows to integrate such approach in IR systems and design systems that are able to account for multiple aspects when returning a ranked list of items to users.
Moreover, this project enhanced the career perspectives of the research fellow, who got a tenure track position at the host university.