The progress beyond the SOTA in the technology are multiple:
1) The delivery of the appliance prototype featuring:
• An in-memory operational database blended with a fast-analytical engine delivering real-time data, able to answer analytical continuous queries over operational data.
• An in-memory data streaming engine integrated with the operational database that provides the framework for parallel analytics algorithms on operational data streams and complemented with a time series analytics library.
• An operational data lake that enables the use of Spark and Hadoop analytics frameworks on operational data.
• A hardware appliance leveraging the BullSequana S server, ranging from 2 to 32 processors (up to 896 cores and 1792 hardware threads), up to 32 GPUs and 48 TB RAM, and 64 TB of non-volatile RAM.
• A management system to deploy and monitor the prototype, based on the Atos Codex AI suite, which provides for the deployment of Platform as a Service (PaaS) and/or Software as a Service (SaaS) cloud on various Infrastructure as a Service clouds, or on HPC clusters.
The prototype provides for On Line Transaction Processing (OLTP), On Line Analytics Processing (OLAP) and on the fly real time analytics on data streams within a single platform.
Moreover, the prototypes exhibit the following non-functional features: • Horizontal scalability, inherited from the built-in features of the components. • Vertical scalability, thanks to the NUMA awareness implemented by the project to the components. • High availability through replication. • Smart placement through efficient resource allocation algorithms. • Dynamic migration across appliances.
2)The components of the optimized software ecosystem, and their performance:
- the scalable in-memory operational database leverages a new Data Engine, a new Query Engine and Transactional Engine. Now it is able to scale over 100 cores, to utilize efficiently over 10s of TB of Memory and to deliver over 100,000 TPM in the OLTP Industrial Benchmark TPC-C.
- the scalable ActivePivot in-memory Analytics brings an innovative architecture benefiting from a custom memory allocator, compressed data structures and NUMA awareness & partitioning. It is already production at several customers where it unlocks things that were not previously possible.
- the extension to Spark to make it NUMA aware showing improvement in terms of duration of the run and CPU consumption. The bigger the input dataset, the higher is the gain. The experimentation of GPU acceleration of PySpark (the Python API of Spark) workflows, using the Nvidia RAPIDS library (Rapids/AI), demonstrated the feasibility of this integrtaion and showed an execution three times faster.
- the scalable Data Streaming Engine brings an innovative optimization in the implementation of the tuple structure, the window processing, and streaming Operators. it is highly available and supports self-reconfiguration. It attained the targeted scalability.
- the parallel incremental sketching approach of ParCorr algorithm proves to be tens of times faster. ParCorr scales up to 100 million time series and is linearly scalable.
- the implementation of the ParCorr algorithm on top of the data streaming engine, and its extension with machine learning capabilities.
3)These technological progresses make it possible to enhance our selected use cases beyond SOTA. For example, for Mobile phone number migration CloudDBAppliance can provide a technological solution for a national central database for cell number portability, capable of providing service to a large country with a number of cell-phone users, of up to 100 million.
