Descrizione del progetto
Soluzioni per un apprendimento distribuito più rapido
L’apprendimento distribuito (DL, distributed learning) si trova dinanzi a due sfide principali da superare: i ritardi causati dai lavoratori più lenti, a cui ci si riferisce con il termine ritardatari, e gli elevati costi di comunicazione associati all’invio di dati di grandi dimensioni. La codifica del gradiente (GC, gradient coding) può aiutare a risolvere i problemi legati ai ritardatari, mentre l’uso di dati a 1 bit riduce i carichi della comunicazione; ciononostante, i metodi attualmente a disposizione non consentono di affrontare entrambi i problemi congiuntamente, soprattutto quando si trasmettono dati a 1 bit, mentre le tecniche per la codifica del gradiente esistenti faticano a funzionare anche con vettori codificati a 1 bit. Sostenuto dal programma di azioni Marie Skłodowska-Curie, il progetto 1-Bit GC-DL affronta questi problemi sviluppando nuovi approcci per l’apprendimento distribuito. Il progetto introduce il metodo GC-DL a 1 bit, che utilizza la codifica del gradiente a 1 bit per gestire i ritardatari e ridurre le dimensioni dei dati; inoltre, un secondo metodo, chiamato LA-GC-DL a 1 bit, abbassa ulteriormente i tempi di formazione selezionando solamente i lavoratori chiave per ogni iterazione.
Obiettivo
In the framework of distributed learning, to mitigate the negative impact of the stragglers on the training time, the gradient coding (GC) technique has been adopted. On the other hand, to deal with high communication burden in distributed learning, 1-bit gradient vectors can be transmitted instead of real-valued ones. However, the existing distributed learning method based on 1-bit data does not take stragglers into account. In addition, current GC techniques are only designed for the distributed learning scheme where real-valued encoded vectors are transmitted and it is difficult to apply them under the case where 1-bit vectors are transmitted.
To overcome the above drawbacks and to reduce the communication overhead and the training time simultaneously, this project aims to propose novel distributed learning methods based on GC with 1-bit data. First, this project will propose a distributed learning method named 1-Bit GC-DL, which develops a 1-bit GC strategy to encode the locally computed gradient vectors of the allocated subsets into 1-bit data. Based on that, the aggregation rule at the central server for the received 1-bit data will be designed, which guarantees that the central server computes an approximated version of the true gradient vector in the presence of a certain number of stragglers to. Second, to further reduce the training time of 1-Bit GC-DL, this project will propose a lazily aggregated distributed learning method based on 1-bit GC, i.e. 1-Bit LA-GC-DL, by combining 1-Bit GC-DL with the lazily aggregated strategy. In 1-Bit LA-GC-DL, only a fraction of the workers participate in local training during each iteration and this project will provide the criterion for selecting the participating workers based on Age of Information. The proposed methods will be compared with other state-of-the-art methods in the context of distributed learning on both simulated and realistic datasets under practical scenarios.
Parole chiave
Programma(i)
- HORIZON.1.2 - Marie Skłodowska-Curie Actions (MSCA) Main Programme
Invito a presentare proposte
(si apre in una nuova finestra) HORIZON-MSCA-2023-PF-01
Vedi altri progetti per questo bandoMeccanismo di finanziamento
HORIZON-TMA-MSCA-PF-EF - HORIZON TMA MSCA Postdoctoral Fellowships - European FellowshipsCoordinatore
100 44 Stockholm
Svezia