A Framework for Auditing Recommendation Engines under the DSA

Opis projektu

Infrastruktura do wdrażania unijnego aktu o usługach cyfrowych

Niedawne pojawienie się dużych modeli językowych (LLM) i sztucznej inteligencji (AI), wraz z ich rosnącą rolą w mediach społecznościowych i wyszukiwarkach (SE), podkreśla potrzebę ich monitorowania i zapewnienia bezpiecznego działania. Wdrożenie unijnego aktu o usługach cyfrowych (DSA) jest kluczowym elementem tych wysiłków. Finansowany przez ERBN projekt FARE_DSA ma na celu opracowanie ram przepływu informacji dla szczegółowego audytu wyszukiwarek i dużych modeli językowych opartych na czatach oraz dostarczenie narzędzia testowego, które będzie wspierać i zachęcać do wdrażania artykułu 40 DSA. Projekt zbada rolę i wpływ algorytmów modeli językowych i wyszukiwarek na procesy wyborcze oraz zapewni praktyczne wskazówki pomagające pokonać wyzwania związane z wdrażaniem zapisów DSA.

Cel

This proposal presents a novel pipeline framework designed to systematically audit Search Engines (SEs) and chat-based Large Language Models (LLMs), serving as a proof-of-concept tool to support the implementation of Article 40 of the EU's Digital Services Act (DSA). The project addresses two critical goals: 1) to investigate how proprietary algorithms in SEs and LLMs influence electoral processes, and 2) to provide practical guidance to overcome complex DSA implementation challenges.

SEs, such as Google, and LLMs, such as ChatGPT, have become primary channels for accessing information. Yet, both rely on opaque and fast-evolving algorithms that are difficult to audit. This raises significant concerns regarding the fairness, transparency, and neutrality of information they offer, particularly in contexts critical to democracy, such as elections: given their widespread use, even minor algorithmic biases can have significant impacts, influencing voter perceptions and electoral outcomes.

The proposed framework leverages resources developed in two ERC-funded projects, including an advanced bot-based auditing tool. Previous studies conducted during high-stakes events, such as the 2024 EU Parliamentary elections, show systematic biases in search results and LLM responses and it is crucial to better understand the origin and implications of these observations. With FARE_DSA we want to probe this systemic bias by combining our independent auditing tool with the first-ever instances of granted access to platform data, through Article 40.

In collaboration with different stakeholders, we will address key technical and procedural hurdles and deliver actionable insight on DSA implementation. Overall, this proposal seeks to establish a rigorous, reproducible auditing methodology capable of informing policy decisions, enhancing transparency, and bolstering democratic resilience through improved regulation and oversight of algorithm-driven platforms.