The project will establish a European standard for the storage and retrieval of multidimensional high-performance computing (HPC) data. It addresses a main technical obstacle, the delivery bottleneck of large HPC results to the users, by augmenting high volume data generators with a flexible data management and extraction tool for spatio-temporal data. The multidimensional database system RasDaMan developed in FP4 will be enhanced with intelligent mass storage handling and optimised towards HPC. A critical mass of large European HPC centres participates; they all will operate the common platform, evaluating it in different HPC fields (University of Surrey: turbulent and combusting flows; CLRC, DKRZ: climate modelling; Cineca: cosmology; DLR: satellite image archives; IHPC&DB: gene expression simulation; CSCS: flow modelling of chemical reactors; Numeca: CFD post processing); the RasDaMan developers provide database skills. The outcome will be a field-tested open prototype platform with flexible, standard based retrieval on multi-Terabyte data in heterogeneous networks. The result will be produced by two SME partners, Active Knowledge GmbH and Numeca International S.A.
The overall objective is to develop a standard architecture and best practice knowledge for the efficient combination of HPC data processing and management, including fast & flexible retrieval, which is accepted as a broad consensus among European HPC centres. The participating HPC centres will act as multipliers to further centres and the European industry using HPC. The architecture will be implemented in various environments and for the most important HPC applications, forming a common pilot platform thoroughly evaluated by end users. Each application field is assigned to a partner with particular experience in this area. Individual objectives of the partners are: - For the vendor partners Active Knowledge GmbH and Numeca International S.A. the outcome significantly enhances their products, increases product visibility, and opens up large new markets. - For the HPC institutions, goal is to strengthen their research position and to offer improved services to their users and clients. - For the database researchers, goal is to strengthen their internationally acknowledged position in multidimensional database technology.
Based on the multidimensional DBMS RasDaMan provided by the partner Active Knowledge GmbH, work is divided as follows: Each HPC partner will couple their HPC data generator and visualisation/post-processing front ends to RasDaMan as the common data management platform. The resulting systems will be operated under real life conditions with end users involved.
Each application area will be assigned to a particular partner with acknowledged experience:
- University of Surrey: simulation and experimentation turbulent and combusting flows;
- CLRC and DKRZ: climate modelling and research;
- Cineca: cosmology;
- DLR: satellite image retrieval and information extraction by post processing;
- IHPC&DB: simulation of the dynamics of gene expression;
- ETHZ flow modelling of chemical reactors
Numeca International S.A. will adapt its software to the DBMS to get into new markets. Active Knowledge GmbH, together with the research lab FORWISS, will extend the DBMS to optimally cope with the multi-Terabyte data sets by adding tertiary storage support and parallelism to the DBMS. Further, these partners will give necessary support to the other partners concerning database organization and operation. ERCOFTAC (European Research Community on Flow, Turbulenc and Combustion) will organise and lead the User Interest Group to raise awareness among further institutions and, particularly, among industry.
The milestones in the project are: M1 (9 months): analysis and specification (report) M2 (18 months): Demonstration of pilot platforms at each HPC lab: pilot platform 1st version; M3 (24 months): pilot platform second version; M4 (30 months): pilot platform 3rd version; user assessment; M5 (36 months): pilot platform final version and applications.
Funding SchemeCSC - Cost-sharing contracts
195251 St. Petersburg
40033 Casalecchio Di Reno (Bologna)
OX11 0QX Chilton, Didcot
GU2 7XH Guildford