Skip to main content
European Commission logo print header

Parallel Grid-aware Library for Neural Networks Training

Final Report Summary - PAGALINNET (Parallel Grid-aware Library for Neural Networks Training)

The aim of this Marie Curie International Incoming Fellowship 221524 “PaGaLiNNeT - Parallel Grid-aware Library for Neural Networks Training” was to develop the enhanced parallel algorithms of neural networks training with improved parallelization efficiency on heterogeneous computational Grids.

The objectives set out at the beginning of the project (for the period 01/04/2009 – 31/03/2011) were as follows:
1. to adapt the BSP cost model of parallel NN training algorithms within single pattern, batch pattern and modular approaches to heterogeneous computational Grid resources of host institution;
2. to develop enhanced single pattern and batch pattern parallel NN training algorithms based on improved communication and barrier functions.
3. to develop a method of automatic matching of parallelization strategy to architecture of appropriate parallel computing system;
4. to develop parallel Grid-aware library for neural networks training capable to use heterogeneous computational resources;
5. to test experimentally parallel Grid-aware library for neural networks training on heterogeneous computational Grid system of host institution within the tasks of one of its active projects;

The main outcomes of the Fellowship are:
• The enchanced parallel batch pattern back propagation training algorithms of multilayer perceptron and recurrent neural network are developed. We have researched the parallelization efficiencies of these parallel algorithms for typical training scenarios with increasing of neural network connections from 36 to 5041 and increasing the number of training patterns from 100 to 10000 on symmetric multiprocessor, multi-core and cluster systems using MPICH2-1.2.1 NEC MPI/EX 1.5.2 and Open MPI 1.4 libraries. Our results show that the developed parallel algorithms are scalable and the parallelization efficiencies are high enough for their efficient use on general-purpose parallel computers and computational clusters;
• The BSP (Bulk Synchronous Parallelism)-based computational cost model of the parallel algorithms is created for the prediction of the expected training time of the neural network on different processors of different parallel systems. The application of this model shows good prediction accuracy, for instance, the maximal absolute error of prediction of parallelization efficiency does not exceed 5% on symmetric multiprocessor computer and 2% on computational cluster;
• The resource brokering strategy is developed based on the training time prediction using BSP-based computational cost model to keep high parallelization efficiency of the neural network parallel training algorithm. Our strategy of resource brokering is based on three criteria, a cost of a parallel system, a predicted execution time and a parallelization efficiency of the parallel algorithm. The strategy of resource brokering is based on Pareto optimality with the weighted sum approach for choosing optimal solutions for efficient parallelization of the algorithm. Our results show good conformity with the desired scheduling policy of minimization of the execution time of the parallel algorithm with maximization of the parallelization efficiency in the most economic way. This approach is founded to be very applicable for the practical implementation of the resource broker when any desired scheduling policy could be described by a user during runtime;
• The parallel Grid-aware library for neural networks training is developed on C programming language and deployed on computational Grid of the host institution and other parallel systems available for the experimental research within the project. The library consists of the C routines of: (i) the enhanced parallel batch pattern back propagation training algorithms of multilayer perceptron and recurrent neural network with advanced intercommunication functions between the parallel parts of the algorithms, (ii) the BSP-based computational cost model of the enhanced batch pattern parallel algorithms, (iv) the coarse-grain modular neural network parallelization strategies of static and dynamic mapping. The resource broker implementation depends on the type of the parallel system used.
• Three actual scientific tasks sufficient to be solved by using neural networks of the team from the host institution were identified and solved jointly with the appropriate scientific groups: (i) stock price prediction for financial markets in a field of finance [16], (ii) classification of adverse events for heart failure patients and (iii) detection of micronucleus in human lymphocytes in a field of medical care. The developed parallel library was successfully applied for speedup the training process of neural networks for the task of stock price prediction for financial markets (with 83% of parallelization efficiency) and for the task of classification of adverse events for heart failure patients (with 95% of parallelization efficiency).