REfactoring Parallel Heterogeneous Resource-Aware Applications - a Software Engineering Approach

Project Information

RePhrase

Grant agreement ID: 644235

DOI

10.3030/644235

Project closed

EC signature date 11 December 2014

Start date 1 April 2015

End date 31 March 2018

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 3 574 027,00

EU contribution

€ 3 574 027,00

Coordinated by

THE UNIVERSITY COURT OF THE UNIVERSITY OF ST ANDREWS
United Kingdom

Periodic Reporting for period 2 - RePhrase (REfactoring Parallel Heterogeneous Resource-Aware Applications - a Software Engineering Approach)

Reporting period: 2016-10-01 to 2018-03-31

Summary of the context and overall objectives of the project

The focus of RePhrase has been on producing new software engineering tools, techniques and methodologies for developing data-intensive applications and targeting heterogeneous multicore/manycore systems that combine CPUs and GPUs into a coherent parallel platform. Data-intensive applications are one of the most important and commonly encountered classes of industrial application. Such applications are often potentially highly parallel. However, it is even harder to obtain good performance, since many additional issues related to data management need to be taken into account.

RePhrase directly targeted ICT 9. It aimed to achieve a breakthrough in simplifying programming parallel data-intensive systems. By using a new RePhrase software development methodology combined with new tools, we aimed to achieve significantly enhanced levels of reliability, robustness, resilience and software integrity compared with conventional approaches. Use of this new methodology and the associated tools will help to foster increased growth and a more competitive EU software industry. The approach will be demonstrated through large-scale applications.

The project has demonstrated that it is possible to achieve good performance for heterogeneous parallel software at low programmer effort,
while maintaining the required standards of software quality. It has led to the development of new tools that have widespread potential use, and new software quality standards for developing multicore software in C++ that are being adopted by the automotive industry.. This will potentially lead to cheaper, more reliable, more robust and more efficient software and systems, capable of exploiting heterogeneous multicore systems.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

In the first period, we identified initial and extended sets of patterns for data intensive parallel programming, based on existing pattern sets (D2.1). We extended the IBM FOCUS tool and the ExpliSAT tool to provide test planning and verification for parallel, patterned, applications. We produced new mechanisms for detecting violations of extra-functional requirements (D3.1). We developed new parallel scheduling and mapping mechanisms, plus performance monitoring (D4.1). We produced requirements capture processes (D5.1) and developed new coding and data standards to support parallel program development (D5.2).

We implemented the initial patterns and a DSL (D2.1). We have designed a new API (D2.4). We have new program shaping techniques for parallelization, (D2.3). We have new refactorings to introduce parallelism (D2.2). We extended IBM’s ExpliSAT to support concurrency, PRL’s QA-Verify tool to support multithreading (D3.1) developed new tooling to detect the violation of extra-functional properties (D3.2) developed static mapping , developed an initial dynamic scheduling toolfor monitoring performance (D4.1).

We have shown how IBM’s ExpliSAT tool can determine functional correctness, PRL’s QA-Verify tool can determine code complies with the coding standards in D3.1; identified requirements on the project use cases for evaluation against the software lifecycle (D6.1).

We identified detailed metrics for robustness etc (D3.1); requirements on the use case applications (D6.1); and selected appropriate use cases based on these requirements (D6.3).

In period 2, we have explored the usability of pattern set D2.7; a new refactoring tool that (D2.6 D2.10); we have new patterns in D2.8. In WP3, we have extended IBM FOCUS and ExpliSAT tool to provide test planning and verification. This work is reported in D3.4 and D3.5. In WP4, we have new parallel scheduling and mapping mechanisms, plus performance monitoring (D4.2 D4.3 and D4.4). In WP6, we have summarized our requirements capture on the use-cases (D6.1 D6.3).

We have developed new pattern discovery techniques and tools D2.9; new refactoring tools D2.6 D2.10; extended PRL’s QA-Verify tool to support new analyses for automatically analyzing code and verifying compliance to standards (D3.3. D3.4) PRL extended their QA-Verify tool to achieve full coverage of the C++17 standard. In WP4, we have developed new software for the adaptivity of pattern applications for the initial pattern set (D4.2 D4.4) and heterogeneous hardware (D4.3). In WP5 we have provided a new inter-operable tool chain combining the different RePhrase tools into the PRL QA-Verify framework (D5.6).

In WP3, we have shown how QA-Verify tool can determine whether code complies with existing C++17 coding standards (D3.4 and D3.5). In WP5, we have developed new requirements capture processes for parallel data-intensive software (D5.1) and new data and coding standards for parallelism (D5.2). This will lead into standardization activities. In WP6, we have evaluated the use-cases against the objectives outlined in Objective 2 (D6.4 D6.5 D6.6 D6.7).

In order to demonstrate robustness, in WP3, USTAN have extended their refactoring tool to deal with safety issues in parallel code bases; UC3M have extended ThreadSanitizer to support the semantics of lock-free data structures; IBM have extended FOCUS CTD and ExpliSAT; these results are reported in D3.5. In WP6 SCCH have worked in defining metrics to evaluate reliability, robustness, adaptivity, resilience and integrity and ways to evaluate them. Furthermore, SCCH have evaluated its use case on the defined metrics and reported the results in D6.6 and D6.7.

In WP5 we have a interoperable tool-chain via QA-Verify tool (D5.6). In WP6 we have evaluated our approach by the use-cases from SCCH, EvoPro, and CiberSam. This is reported in D6.4 D6.5 D6.6 and D6.6.

The project website provides technical reports + tutorials (D7.6). 126 research papers, 5 workshops and 1 tutorial. Our twitter has 500+ followers (D7.8). Research papers are available. Project data is on Zenodo (D7.3). We have press releases (D7.7).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

• introducing a new software engineering methodology;

• building on our existing pattern-based programming methodology to assist the design and implementation of parallel data-intensive software;

• developing automated tools for tuning and deploying applications on a wide range of different heterogeneous hardware platforms, in a range of deployment settings;

• developing new methods for discovery of parallel patterns in existing sequential C++ code and reshaping existing C++ code to prepare it for patterns;

• developing new methods and tools for testing and verification of requirements for data-intensive applications;

• developing refactoring tools for introducing, rewriting and tuning parallel patterns in both new and existing C++ applications;

• integrating tools that address all aspects of the software development process into a coherent software engineering methodology for data-intensive applications, and ensuring their inter-operability;

• defining standards for programming pattern-based data-intensive applications, and developing new tools for automatically checking compliance with these standards.

RePhrase results provide significant and demonstrable productivity increase in the development, testing, verification, deployment and maintenance of parallel systems. This impacts data-intensive systems (which are often naturally parallel), and distributed/cloud systems. Productivity improvements are verifiable through reductions in coding errors and bugs, improvements in robustness, and increased portability and evolution of the software code.

Overview of Project Objectives