Skip to main content
Go to the home page of the European Commission (opens in new window)
English en
CORDIS - EU research results
CORDIS

Adaptive Tokenization and Memory in Foundation Models for Efficient and Long-Horizon AI

Project description

AI models that work smarter, not harder

As AI models become larger and slower, the challenges of environmental impacts and energy demand continue to grow. Current models process and memorise information as a sequence of representations with a fixed granularity. The ERC-funded AToM-FM project plans to create more efficient AI models by adapting the length of each sequence on the fly, without discarding information. By overcoming the main bottleneck of current model architectures, AToM AI models will be faster and leaner, leading to energy saving and reduced environmental harm. AToM-FM will unlock permanent memories, allowing AI models to learn continuously, and enhance reasoning capabilities, allowing AI models to explore more complex hypotheses for the same reasoning time. The project will usher in an era of smarter and greener AI technology.

Objective

"The recent revolution in generative AI is powered by the ever-growing scale of Foundation Models (FMs). This, however, causes a series of harmful ramifications, such as their unsustainable energy demand and environmental pollution, which accelerate climate change. Moreover, the scale of FMs jeopardises data privacy, as it compels users to deploy them on third-party servers rather than edge devices. AToM-FM sets out to reverse this trend by remedying a fundamental source of inefficiency in FMs: the granularity of the ""atomic"" units for representing information in current FMs is fixed, as it entirely depends on how they update their memory and segment input data (a process known as tokenization). Instead, AToM-FMs will couple their granularity with the complexity of each task, allocating only as much computing effort as needed. As its key technical breakthrough, AToM-FM will make memory and tokenization adaptive, elevating FMs to unprecedented levels of efficiency. To facilitate the swift adoption of this new technology, I will repurpose state-of-the-art FMs into adaptive architectures and release them to the public. This will not only drastically cut the carbon footprint of FM deployment and uphold data privacy, but also foster the emergence of new FM capabilities. In fact, compressing the FM representations will effectively broaden their horizon, i.e. the amount of input and output they can perceive and generate, respectively, without exhausting their memory. This will unlock 1) permanent memories, which are a prerequisite for lifelong learning; 2) longer reasoning chains, perfecting the mathematical and planning skills of FMs; 3) modality-agnostic and fine-grained inputs, facilitating the integration of low-resource modalities and grounding agentive FMs onto physical spaces. Overall, AToM-FM's ambitious agenda will usher in the next generation of FMs, paving the way for efficient and long-horizon AI."

Fields of science (EuroSciVoc)

Keywords

Programme(s)

Topic(s)

Funding Scheme

HORIZON-ERC - HORIZON ERC Grants

See all projects funded under this funding scheme

Call for proposal

(opens in new window) ERC-2025-STG

See all projects funded under this call

Host institution

IMPERIAL COLLEGE OF SCIENCE TECHNOLOGY AND MEDICINE
Net EU contribution
€ 1 418 809,59
Address
SOUTH KENSINGTON CAMPUS EXHIBITION ROAD
SW7 2AZ London
United Kingdom
Region
London Inner London — West Westminster
Activity type
Higher or Secondary Education Establishments
Links
Total cost
€ 1 418 809,59

Beneficiaries (2)