Machine Learning for Offensive Computer Security

Informacje na temat projektu

MALFOY

Identyfikator umowy o grant: 101043410

DOI

10.3030/101043410

Data podpisania przez KE 15 Września 2022

Data rozpoczęcia 1 Stycznia 2023

Data zakończenia 31 Grudnia 2027

Finansowanie w ramach

• European Research Council (ERC)

Koszt całkowity

€ 1 962 000,00

Wkład UE

€ 1 962 000,00

Koordynowany przez

TECHNISCHE UNIVERSITAT BERLIN
Germany

Periodic Reporting for period 1 - MALFOY (Machine Learning for Offensive Computer Security)

Okres sprawozdawczy: 2023-01-01 do 2025-06-30

Podsumowanie kontekstu i ogólnych celów projektu

Machine learning has been a driving force behind recent advances in science and industry. In particular, deep neural networks have enabled remarkable progress in solving challenging problems, such as natural language translation and image content recognition. This success has sparked a wave of research on the use of machine learning in computer security. Numerous approaches have emerged that leverage learning models to detect network intrusions, analyze security events, and understand malicious software. However, this research has focused almost exclusively on the defender’s perspective—developing methods for detecting attacks and remediating security issues. This perspective is incomplete: Security is inherently adversarial, marked by a continuous cycle in which attackers and defenders compete. As such, it is insufficient to study defenses in isolation. A comprehensive understanding of future threats requires incorporating the offensive perspective into the picture.

Surprisingly, little research has investigated the role of machine learning in offensive security. The simple question “How would a hacker use machine learning?” remains largely unanswered. While the benefits of learning algorithms in defense are well understood, their potential as offensive tools is still vague and underexplored. This gap hampers our ability to anticipate emerging threats and to build robust defenses. In reality, offensive techniques already play a critical role in cybercrime, generating billions of dollars in illicit profit each year. It is only a matter of time before malicious actors adopt machine learning at scale to carry out more effective attacks.

The project MALFOY aims to close this gap by systematically exploring how learning algorithms can be used in offensive security. By adopting the perspective of the attacker, we seek to understand how offensive tasks can be accelerated and automated using modern learning paradigms such as deep generative models and deep reinforcement learning. To this end, we develop interfaces between offensive techniques and learning algorithms, enabling these techniques to improve autonomously in tasks like reconnaissance, vulnerability discovery, and attack generation.

This new integration opens the door to fundamentally new protection strategies. By anticipating the capabilities of intelligent offensive tools, we can devise disruptive defense mechanisms that stay one step ahead of evolving threats. The project thus paves the way for a new understanding of the interplay between computer security and machine learning. We expect it will give rise to novel methods for constructing attacks and assessing system security. These methods can be used proactively to detect and eliminate vulnerabilities before they are exploited. At the same time, insights into future attack strategies form a foundation for building long-term and resilient defenses.

Prace wykonane od początku projektu do końca okresu sprawozdawczego oraz najważniejsze dotychczasowe rezultaty

During the first reporting period, the project developed novel techniques for proactively identifying and mitigating security vulnerabilities, with findings published in top security conferences. The research was structured along the four work packages of the project.

In WP1 (Target Reconnaissance), the project analyzed how adversaries can automatially explore network environments, focusing on email security. A large-scale study of SPF, a key security mechanism in emails, across 12 million Internet domains revealed security gaps in approximately 150,000 domains, leading to notifications to all affected operators (IMC 2023).

In WP2 (Vulnerability Discovery), the project applied machine learning to identify vulnerabilities in software, resulting in three key publications. The first introduced a novel approach for detecting defects in JSON parsers through differential testing (AsiaCCS 2024). The second uncovered confounding effects in learning-based vulnerability discovery, demonstrating flaws in prior research (AISEC 2023). The third provided the first systematic comparison of target selection strategies for directed fuzzing, evaluating the strengths and limitations of learning-based approaches versus traditional techniques (AsiaCCS 2024).

In WP3 (Exploit Preparation), the project investigated how machine learning can enhance exploit development. Initial experiments on program memory did not produce the anticipated results, prompting a shift toward alternative directions. One focus area explored binary code analysis, demonstrating that end-to-end machine learning is a powerful approach for code analysis, in contrast to assumptions made in prior work (AsiaCCS 2024). Another line of research examined the attribution of malicious code, showing that while attackers may attempt to evade identification, attribution remains effective when defenders have access to sample code from adversaries (PETS 2023).

In WP4 (Attack Construction), the project finally explored machine learning-driven attack strategies, with a particular focus on web security and PDF-based attacks. The first major contribution, developed in collaboration with TU Braunschweig, introduced a novel approach to cross-site scripting attacks, enhancing the automation of exploit triggers (USENIX Security Symposium 2024). The second contribution demonstrated how PDF manipulation can be used to deceive academic conference systems and the reviewing process (USENIX Security Symposium 2023).

All research was conducted in strict adherence to the project's ethical guidelines, with oversight and consultation from its ethical advisory board. All identified vulnerabilities and weaknesses have been reported to the affected parties prior to any publication.

Innowacyjność oraz oczekiwany potencjalny wpływ (w tym dotychczasowe znaczenie społeczno-gospodarcze i szersze implikacje społeczne projektu)

During the first reporting period, the project could advance the state of the art in security research through different contributions. One major achievement was the first fully automated method for generating cross-site-scripting polyglots for web attacks, replacing traditionally manual efforts. This advancement improves both vulnerability detection and mitigation, leading to a more systematic approach to security analysis of web applications. The significance of this work was recognized with the Distinguished Paper Award at the USENIX Security Symposium 2024.

Another important result established theoretical foundations for the anonymization of source code, proving its inherent difficulty. The research formally demonstrates the challenges of making source code untraceable, contributing to the development of more effective attribution techniques for identifying malicious actors as well as understanding the problem of publishing software anonymously. These findings provide valuable insights for improving security and privacy for source code analysis.

Finally, an unexpected but notable outcome was the discovery of vulnerabilities in the academic peer-review process. The project demonstrated how machine learning can be exploited to manipulate the automated paper-reviewer assignment in the review process, exposing new attack surfaces in AI-driven decision-making. Beyond its technical implications, this work highlights emerging security risks resulting from the integration of AI and underscores the need for stronger safeguards in automated decision-making systems.