Decision-theoretic sequential decision making (SDM) is concerned with endowing an intelligent agent with the capability to choose actions that optimize task performance. SDM techniques have the potential to revolutionize many aspects of society, and successes such as beating a grandmaster in the game of Go, have sparked renewed interest in this field. However, despite these successes, fundamental problems of scalability prevent these methods from addressing other problems with hundreds or thousands of state variables. For instance, there is no principled way of computing an optimal or near-optimal traffic light control plan for an intersection that takes into account the current state of traffic in an entire city.
INFLUENCE aimed to develop a new class of influence-based SDM methods that overcome scalability issues for such problems by using novel ways of abstraction. For instance, the intersection’s local problem is manageable, but the influence that the rest of the network exerts on it is complex. The key idea that we explored is that by using machine learning methods, we can learn sufficiently accurate representations of such influence to facilitate near-optimal decisions. We call these representations 'approximate influence points' (AIPs).
The objectives were to
1 generate formal understanding of the use of AIP representations
2 develop methods that can induce representations for AIPs
3 develop novel simulation-based planning methods that use AIPs to efficiently plan for very large problems,
4 develop novel influence-based reinforcement learning (RL) methods
5 investigate approaches to exploit AIPs in multiagent coordination.
These objectives have been realized to a large extent:
We published 42 papers, 23 of which in top-tier AI and ML venues. We have further developed the framework of influence-based abstraction and provided conditions under which we can have bounds on the quality of AIPs. As part of a number of papers we have learned effective AIP representations, and we demonstrated that these can improve the efficiency of several tasks. Specifically, we demonstrated that we can improve the task performance of complex online planning problems with 100s of variables. For even more complex problems, we have shown that AIPs can lead to more efficient deep RL significantly reducing the time needed to train without reducing quality, and that they can also improve multiagent learning by parallelizing learning in different sub-problems.
In this way, INFLUENCE has made an important step towards realizing the promise of autonomous agent technology, particularly for domains with local effects of the actions of agents, such as intelligent traffic light control, or coordination of multi-robot teams.
