Self-Adjusting Networks

Project Information

AdjustNet

Grant agreement ID: 864228

DOI

10.3030/864228

Project closed

EC signature date 20 December 2019

Start date 1 March 2020

End date 30 November 2025

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 670 822,99

EU contribution

€ 1 670 822,99

Coordinated by

TECHNISCHE UNIVERSITAT BERLIN
Germany

Periodic Reporting for period 3 - AdjustNet (Self-Adjusting Networks)

Reporting period: 2022-11-01 to 2024-04-30

Summary of the context and overall objectives of the project

This project revolves around communication networks which have become a critical infrastructure of our digital society. Due to the explosive growth of data-centric applications and the resulting increasing workloads headed for the world’s datacenter networks, today’s static and demand-oblivious network architectures are reaching their capacity limits.

The AdjustNet project pursues a radically different perspective, envisioning demand-aware networks which can dynamically adapt their topology to the workload they currently serve. Such self-adjusting networks hence allow to exploit structure in the demand, and thereby reach higher levels of efficiency and performance. The AdjustNet project is timely and enabled by recent innovations in optical communication technologies which allow to flexibly reconfigure the physical network topology.

With the AdjustNet project we aim to lay the theoretical foundations for self-adjusting networks. To this end, our goal is to identify metrics that serve as yardstick of what be achieved in a self-adjusting network for a given demand and to devise efficient algorithms for online network adaption. We further aim to understand to which extent network operations in general can be automated, which requires giving away control, similar to self-driving cars. We also aim to complement our foundational research with case studies, exploring novel datacenter architectures.

If successful, self-adjusting networks can have a large impact on society. In particular, the project can enable our digital society’s backbone networks to serve the next generation’s data-intensive workloads, at unprecedented levels of efficiency, performance, and reliability. Forecasts predict a fast growth of annual energy consumption and global CO2 emission due to cloud computing, up to 1/5th of the Earth’s power consumption by 2025. Thus, already a small improvement in utilization, as achieved with self-adjusting networks, may reduce costs and energy consumption significantly. Furthermore, the ground-breaking concepts and techniques developed in this project may also lead to insights which can in turn influence technological developments. Given that the community is currently re-architecting the networking stack anyway, we believe that now is the time we can have the most impact by bridging the gap between theory and practice.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

With the ERC funding, we managed to put together a very strong team of researchers, including PhD students, Postdocs and senior researchers, and also extended our international collaborations. We are very happy with the progress achieved so far. In particular, regarding WP1, we managed to develop a model for self-adjusting networks which captures the essence of recent datacenter technologies; this model provides a tractable algorithmic abstraction and we were also invited to present it at different companies. In particular, this model allowed us to analytically uncover a mismatch between specific traffic patterns and the datacenter topologies on which they are served. We quantified the resulting performance loss and proposed a novel design, Cerberus, which provably provides an optimal matching. These results were presented at ACM SIGMETRICS 2020 and 2022, and constitute our main contribution to WP2 and WP3. We also made good progress in understanding the design of more robust self-adjusting networks (WP4), and were able to show design a statically optimal self-adjusting network design, ReNet, which provides redundancy leveraging a hybrid design. We are slightly ahead of schedule regarding WP5 and WP6, also thanks to a collaboration with Overseas. In particular, we decided to focus, as a case study, on datacenter networks, and started to explore how to deal with dynamic datacenter topologies on the transport layer. To this end, we developed a novel transport protocol, PowerTCP, which achieves much more fine-grained congestion control by adapting to the bandwidth-window product (“power”). PowerTCP leverages in-band network telemetry to react to changes in the network instantaneously without loss of throughput and while keeping queues short. We further gained important insights into a key challenge of dynamic networks, buffering. Today’s network devices use a hierarchical packet admission control scheme where first, a buffer management scheme decides the maximum length per queue at the device level and then an active queue management scheme decides which packets will be admitted at the queue level. We show that a joint optimization of the two, i.e. an active buffer management (ABM) can significantly improve flow completion times. PowerTCP was published at USENIX NSDI 2022 and ABM at ACM SIGCOMM 2022.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

There are several fundamental questions which we could not solve yet, despite first significant efforts. In particular, we do not understand well yet how to deal with dense traffic, both on an algorithmic level as well as in terms of the achievable metric: while we believe that information-theoretic notions such as the entropy capture well the achievable performance, we could not show this analytically yet. But expect to have such a complete characterization by the end of the project. Furthermore, we yet have to systematically address scalability challenges, which will be important for the success of the project. While our current solutions scale to today`s networks, in the future, they may be insufficient and introduce bottlenecks. Accordingly, we expect to contribute novel results on scalable self-adjusting networks which support a more decentralized operation, also in terms of routing, by the end of the project. Last but not least, we started contributing towards the reproducibility of research on self-adjusting networks: reproducing experiments in existing literature is often difficult due to missing hardware and publicly available software. We started developing a flexible framework for reconfigurable networks based on off-the-shelf hardware, which supports experimentation and reproducibility at a small scale. Our framework can be instantiated with different configurations, allowing us to emulate and study existing and new architectures. We will continue extending this framework and hope that by the end of the project, it can also serve us as a demonstrator, which will be useful to disseminate various of our results as well.

traces.png