Universal Data Compression with Circular Context Trees

Projektinformationen

UNIDACT

ID Finanzhilfevereinbarung: 101158232

DOI

10.3030/101158232

Projekt abgeschlossen

EK-Unterschriftsdatum 13 Dezember 2024

Startdatum 1 Januar 2025

Enddatum 30 Juni 2026

Finanziert unter

• European Research Council (ERC)

Gesamtkosten

Keine Daten

EU-Beitrag

€ 150 000,00

Koordiniert durch

UNIVERSITAT POLITECNICA DE CATALUNYA
Spain

Periodic Reporting for period 1 - UNIDACT (Universal Data Compression with Circular Context Trees)

Berichtszeitraum: 2025-01-01 bis 2026-06-30

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Data compression is everywhere: every phone, computer, car and data centre relies on it to store and move information. Yet the universal algorithms underpinning most of today's systems were devised decades ago — Lempel-Ziv (ZIP) in 1977, the Burrows-Wheeler transform (BZIP), prediction by partial matching (PPM) and context-tree weighting (CTW) between the 1980s and 1990s — with little progress since. These methods are provably optimal only asymptotically, as file length tends to infinity; on the finite files real systems handle, they leave significant room for improvement, even as data volumes and the cost of compressing them inefficiently keep growing. In some settings the limitation is stark: deep-space missions such as Gaia discard valuable observations because on-board storage cannot keep up.

Within the ERC Consolidator Grant ITUL we developed UNIDACT, a universal lossless compression algorithm based on circular context trees that compresses any type of data and, on finite files, outperforms every state-of-the-art alternative tested — commercial (ZIP, BZIP) and academic (PPM, CTW) — by at least 82% on average (over 90% in most scenarios), while keeping linear decoding complexity.

The overall objective of this Proof of Concept was to move UNIDACT from a binary prototype towards exploitation. Technically, this meant generalising the algorithm to arbitrary alphabets and delivering fast C implementations, developing an indexed version (iUNIDACT) giving access to parts of a file without full decompression, and building customised versions for high-value applications. On the business side, the project aimed to produce a market analysis and a software-licensing strategy.

The expected impact follows directly: more efficient universal compression makes information systems cheaper and greener — longer battery life, less storage, faster transfers — with concrete gains where they matter most: recovering otherwise-discarded space-mission data, handling large genomic datasets, and enabling indexed access for cloud and distributed storage. Beneficiaries range from operating-system vendors, still shipping the 1977 ZIP algorithm, to chip manufacturers.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

The technical work built on the binary UNIDACT prototype from the ITUL grant, previously only a memory-heavy MATLAB proof of principle. All planned tasks across the work packages were carried out; the main pieces of work were:

First, a fast C++ implementation of the binary UNIDACT: removing the MATLAB overhead gave a working encoder/decoder with the speed and memory footprint to compare against commercial compressors on equal terms, not only in simulation.

Second, we derived UNIDACT for non-binary alphabets and extended the fast C++ implementation to this case, at similar speed and per-bit memory efficiency. Moving from bits to arbitrary alphabet sizes is the prerequisite for compressing structured data more effectively — ASCII text and the quaternary alphabet (A, T, G, C) of genomic data — and required generalising the circular-context-tree model and its cost/gain computations to arbitrary cardinality.

Third, we made significant progress in the indexed version (iUNIDACT) and the customised genomic and deep-space variants; both require further research — though the general arbitrary-alphabet version can already effectively be applied to those data types today.

In parallel, a market and commercialisation study was carried out (outsourced to Drop-Innovation), covering the market context, technology-transfer pathways and investment thesis, and synthesising the market-entry decision matrix and strategic conclusions into an executive summary informing resource-allocation and commercialisation decisions.

The grant also partly supported related theoretical work on the foundations of source coding, with PoC co-investigator M. Dabirnia — the line from which UNIDACT emerged — published as M. Dabirnia, H. Joudeh and A. Guillén i Fàbregas, "Dual-Domain Expurgated Error Exponents for Source Coding With Side Information," IEEE Trans. Inf. Theory, vol. 72, no. 5, pp. 2689–2703, May 2026, with conference versions (ISIT 2026, Guangzhou; IZS 2026). Co-author H. Joudeh (TU Eindhoven) is himself an ERC Consolidator grantee.

Outcomes of the action:
- Fast, memory-efficient C++ implementation of binary UNIDACT and testing.
- Derivation and C++ implementation of UNIDACT for arbitrary (non-binary) alphabets and testing against standard copruses.
- Foundational work on the indexed version (iUNIDACT) and customised genomic/deep-space variants (requiring further research).
- A market and commercialisation study (context, tech-transfer pathways, investment thesis, executive summary) informing exploitation.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

The project carried out all planned work: fast C++ implementations of UNIDACT (binary and arbitrary-alphabet), the non-binary derivation, initial work on iUNIDACT and the customised variants, and a Drop-Innovation market study sizing the opportunity and route to market.

Market context and potential impact. The study frames the impact against a global datasphere projected to grow from ~175 zettabytes in 2025 to over 1,000 by 2030 while storage covers under 15% of data generated — so compression is increasingly a hard economic constraint. It also sharpens where UNIDACT's value lies: general-purpose compression is commoditised by open-source codecs (Zstandard, Brotli) and offloaded to dedicated silicon, so competing as "a better ZIP" is not the play. Instead, the opportunity concentrates in high-value niches where legacy algorithms are structurally insufficient and where UNIDACT's three differentiators — linear decoding complexity, arbitrary-alphabet support and indexed retrieval (iUNIDACT) — map onto quantifiable pain:

- Cloud data lakes and FinOps. Cloud egress fees are a ~$43 billion/year "tax"; iUNIDACT's retrieval of parts of a file without full decompression attacks that cost directly — the largest single segment in the study.
- Genomics and life sciences. Petabyte-scale sequencing data in a quaternary (A, T, G, C) alphabet is poorly served by binary codecs; the non-binary version targets integration into the CRAM / GA4GH ecosystem.
- Aerospace and deep-space telemetry. Onboard storage and transmission limits force deletion of scientific data; a low-memory, high-ratio codec addresses the ESA/CCSDS need directly.
- Edge and IoT. Linear complexity and small-file efficiency suit battery- and bandwidth-constrained sensors, with a route to per-unit semiconductor-IP royalties.

Significance. On a bottom-up US/EU basis, the study estimates the immediately obtainable market (SOM, years 1–4) at about $19 million ARR — cloud FinOps ~$10M, genomics ~$4.35M edge/IoT ~$4.5M — scaling to a serviceable market (SAM) of ~$555 million and a total addressable market (TAM) of ~$7 billion, plus a further ~$14 billion if UNIDACT is licensed as semiconductor IP. These are the study's estimates, and the SOM is contingent on iUNIDACT reaching production readiness, which requires the further research identified below.

Key needs to secure uptake. The study's central conclusion matches our own: the binding constraint is no longer the technology but the commercial and institutional framework around it. Adoption requires:

- Further R&D to mature the results already advanced: above all iUNIDACT (the gating item for the largest, cloud, segment) to production quality, plus the customised genomic and deep-space variants — already usable today via the general-alphabet version — and a hardware/FPGA path against silicon-accelerated legacy codecs.
- Demonstration and validation on real data, beyond simulated Markov sources, with GA4GH and ESA/CCSDS partners.
- Standardisation as the distribution channel — in genomics (GA4GH/CRAM) and space (CCSDS), adoption runs through standards bodies rather than software sales.
- An IP and commercialisation vehicle — UNIDACT is not patented (software is not patentable at the EPO); it would be protected via proprietary implementations and trade secrets and taken to market through a dedicated spin-off, with a segment-specific licensing model (enterprise licences; SaaS/savings-share for cloud; IP royalties for silicon; NRE for space).
- Access to markets and finance — follow-on and national/EU funding to reach production, while the timing windows the study identifies in genomics standards, aerospace procurement and European funding remain open.

Overview of the results:
- Fast, memory-efficient C implementation of binary UNIDACT.
- Derivation and C implementation of UNIDACT for arbitrary (non-binary) alphabets.
- Foundational work on iUNIDACT and customised genomic/deep-space variants (further research needed).
- Market and commercialisation study (Drop-Innovation): competitive context, four-segment segmentation, TAM/SAM/SOM sizing (~$7B / ~$555M / ~$19M ARR), technology-transfer pathways and investment thesis.