The result of the FEXFEM project has been a scientific computing library for frontier simulations at extreme scales. The library will be distributed as open source software, in order to increase its usage by the scientific community and industry. FEMPAR combines discretization algorithms for problems governed by partial differential equations, using the most advanced finite element techniques, with some of the most scalable linear solvers worldwide. This strong coupling between discretization and linear solver modules allows one to express algorithms in code that are hard to be implemented with weaker coupling (staggered) approaches. FEMPAR has unique advanced features that are hard to obtain in other numerical libraries: nonlinear preconditioners, space-time solvers, and an asynchronous parallel execution model with excellent scalability properties. Based on these capabilities, very large problems have been solved on structured and unstructured meshes, up to 10^11 degrees of freedom on half a million cores. FEMPAR is included in the high-Q club of the most scalable European codes in Europe.
In the frame of this PoC project, we have worked on different tasks in order to simplify the usage of FEMPAR. During the project, we have re-engineered vast parts of its object-oriented software architecture (using, among others, recurrent OO design patterns), in order to simplify user drivers and increase the capabilities of the library. Now, the user can state the problem to be solved in a very easy way, close to how he/she expresses it mathematically on the blackboard. Further, FEMPAR provides advanced discretization schemes like Raviart-Thomas and Nedelec finite elements of arbitrarily high order for electromagnetic and subsurface modelling. On the other hand, we have now drastically simplified the compilation step of the library using CMAKE, and we have developed a CTEST-based system to perform automatic testing. Another important feature in this library are the parallel I/O capabilities for addressing the large scale data generation scalability challenge. In this sense, we have created a new extensible layer for handling the generation of data files for later visualization of results using, e.g. Visit or Paraview. This new layer currently supports VTK and XDMF+parallel HDF5 file formats, although many more can be implemented given its extensible software design.
The project uses Git for version control. The GitLab web-based platform has been used for integrated and collaborative development, including issue trackers, a wiki with guidelines and documentation for developers, merge request code review, continuous integration and testing, CDash analysis of testing results, etc.. We have also made publicly available on GitHub's repositories some of the support and stand-alone software libraries that have been developed in the frame of the project. A GitLab-based webpage will be finished soon, when FEMPAR will be released as open software community project. In order to make easier the learning curve of the library both at the developer and user level, we provide a set of drivers that serve as tutorials. The tutorials show how to use the different capabilities provided by the library. Two journal articles describing the software project in detail (including its software architecture, implementation details and recommendations, examples, use cases, etc.) are expected to be published in the following months.
FEMPAR is now being used in collaboration with some enterprises with HPC simulation requirements. First, we have developed a new module for high temperature superconductor simulations together with Oxolutia (Spain). Second, we have now created a new module for metal additive manufacturing simulations with STAM (Italy).
