Sustainable Training of Code Language Models through Data Refinement

Informazioni relative al progetto

condenSE

ID dell’accordo di sovvenzione: 101151798

DOI

10.3030/101151798

Progetto chiuso

Data della firma CE 6 Marzo 2024

Data di avvio 1 Aprile 2024

Data di completamento 31 Marzo 2026

Finanziato da

• Marie Skłodowska-Curie Actions (MSCA)

Costo totale

Nessun dato

Contributo UE

€ 210 911,04

Coordinato da

SIMULA RESEARCH LABORATORY AS
Norway

Periodic Reporting for period 1 - condenSE (Sustainable Training of Code Language Models through Data Refinement)

Periodo di rendicontazione: 2024-04-01 al 2026-03-31

Sintesi del contesto e degli obiettivi generali del progetto

Large Language Models (LLMs), like ChatGPT, have garnered significant media attention and attracted a growing user base. These models can be trained on source code from platforms like GitHub, enabling them to acquire both semantic and syntactic structure of code. As a result, they can assist with various Software Engineering (SE) tasks, ultimately saving software developers valuable time on laborious tasks, such as bug fixing and code writing. Such Code Language Models (CLMs) rely on extensive pre-training corpora, but the sheer scale of these data and models leads to prolonged training times and high energy consumption.

The computational costs associated with training state-of-the-art Machine Learning (ML) and Deep Learning (DL) models increased by a factor of 300,000 between 2012 and 2018, a concerning trend highlighted in various studies. More recent models, like BLOOM, trained on 46 natural and 13 programming languages required more than a million GPU hours, 433,196 kWh of energy, and produced 81 tons of CO2. This surge in computational costs poses challenges for researchers with limited computational resources, and also raises environmental concerns due to the associated CO2 emissions.

Therefore, it is imperative to assess ML model quality and performance not only on metrics such as accuracy, but also on energy consumption. Techniques to improve training efficiency include quantization, model pruning, algorithm optimisation and dataset reduction. Consequently, this project studies a more sustainable training of CLMs following efficient training paradigms, specifically with data refinement techniques, while maintaining competitive performance on SE tasks.

Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti

The main focus of our research activities was the study of LLMs for their application in program repair and code generation.
Our results show that by using only a fraction (<1%) of the fine-tuning dataset, we can achieve improvements of up to 78% in the number of programs repaired.
Further, using parameter-efficient fine-tuning paradigms such as LoRA allowed us to reduce the number of trainable parameters to less than 1% of standard fine-tuning and improve task performance. Another angle on reducing costs for fine-tuning LLMs is quantization. In our experiments, we observed that we are successful in reducing memory requirements for training while maintaining a comparable level of performance to full precision models. However, inference time and energy consumption are both increased

Further studies in the software engineering domain include a comparative study of LLMs for log parsing, the collection of a novel dataset for coding tasks, and the proposal of different strategies for supporting program repair and defect detection tasks (e.g. ensembles, semantic-preserving transformations, translation).

We proceeded in conducting literature reviews, to support researchers and practitioners in gaining a better overview of existing works and understanding of the current state of research with regards to sustainability and LLMs.
First, we carried out a meta survey which collected 984 surveys on LLMs, eight of which are covering data selection strategies.
Proceeding, we studied the practices of SE researchers when carrying out empirical research with LLMs. This includes the choice of models (open source vs. closed source) and awareness of associated costs when running experiments. Among others, 10% of the studies mentioned monetary costs but no one considered environmental aspects, such as energy consumption or CO2 emissions.
To better understand how SE can aid in supporting sustainability, we conducted a mapping study of the past 15 years of SE research which considered energy. We found 390 publications which addressed fields such as mobile applications or AI, and frequently proposed optimization approaches.

Concurrently, I contributed and continued with work on ML ethics, specifically software fairness. In this regard, two bias mitigation methods (FairST and FairRF) have been proposed. FairST is the first approach based on the principle of Fair Representation Learning devised for post-processing bias repair. FairRF applies multi-objective evolutionary search to optimize for fairness and accuracy. In particular, the best hyperparameter configurations and data mutation are targeted.

Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora)

Eleven publications were produced as part of this project, contributing to the extension of the state of the art as follows:
- A novel approach for program repair which uses round-trip translation (i.e. translation buggy code to an intermediate representation and back). This approach was able to produce 46 unique patches that fine-tuned LLMs were not able to generate.
- A novel dataset of 288,617 “hacks” which are able to make seemingly correct solutions to programming problems fail.
- A comparative study of LLMs for the task of log parsing, showing that open-source models are able to achieve competitive performance when compared to paid, closed source LLMs.
- A collection of 94 semantic-preserving transformations for source code and replication of 39 of them. Our investigation showed that more than half of them were actually not able to preserve semantics.
- An evaluation of parameter-efficient fine-tuning strategies for program repair, achieving better performance with less than 1% of trainable weights.
- Two novel bias mitigation methods that significantly improve fairness over base classifiers and achieve competitive performance against existing methods.
- A collection and overview of existing surveys on LLMs. This includes 871 novel surveys that have not been covered by two existing collections.

Where possible, we make the source code and experimental artifacts available, such that our approaches, datasets and results can be replicated and reused.