The unprecedented evolution of computing power, combined with the decreasing costs of computation and storage infrastructure, has revolutionized all scientific fields and enterprises; at the heart of this revolution is the ability to collect unprecedented amounts of data. Real progress, however, depends on how efficiently we can ‘extract value from chaos’, i.e., process the collected data and transform it into useful information. Unfortunately, despite the impressive growth of data management technologies, data analysis application efficiency is hindered by the increasing data growth. The typical business intelligence architecture is far too complex to yield answers productively, with businesses operating large numbers of specialized data processing systems, with data being repeatedly moved, copied or transformed between them. The reason is that, despite tremendous progress, data management tools are based on legacy designs whose requirements are no longer adequate. A novel approach is needed urgently, or users risk losing the ability to leverage their hard-earned data.
In this proposal, we identify assumptions that do not scale with today’s massive data explosion, and design a comprehensive end-to-end solution that implements a novel form of data virtualization to vastly simplify data analysis. Our approach removes barriers to data handling and maximizes efficiency for science, businesses, and their users, by enabling new forms of data computation that are unrestrained by how data is collected or stored. This project constructs the building blocks for data-driven computing and the fourth paradigm of scientific discovery, thereby advancing our ability to automatically explore massive and complex datasets at sight, and dictate the new developments in sciences and the industry.