Objective 1: Representation Learning. Devise models and algorithms for learning neural belief representations.
a. We developed contrastive learning approach for learning neural belief representations, establishing the theory and practice of this approach. [Choshen and Tamar, ContraBAR: Contrastive Bayes-adaptive deep RL. ICML 2023]
b. We developed the deep latent particle (DLP) structured image representation, a breakthrough that uses image keypoints as latent variables in an image representation. This allows to represent the intricate interactions between several objects in images and video, and we have used it to obtain state-of-the-art results in object-centric image generation, video prediction, and reinforcement learning. [Daniel and Tamar. Unsupervised image representation learning with deep latent particles. ICML 2022; Daniel and Tamar, DDLP: Unsupervised object-centric video prediction with deep dynamic latent particles. TMLR, 2023; Haramati et al., Entity-centric reinforcement learning for object manipulation from pixels. ICLR, 2024]
Objective 2: Scale Up the Framework of Bayesian RL Using Deep Learning.
a. We initiated the theoretical study of Meta-RL, focusing on the question of how much training domains are required to guarantee near Bayes-optimal learning. This investigation allowed us to make progress in a principled approach to offline meta RL, and furthermore, opened a new theoretical research direction that we did not anticipate [Rimon et al., Meta reinforcement learning with finite training tasks – a density estimation approach. NeurIPS 2022; Mutti and Tamar, test-time regret minimization in meta reinforcement learning. ICML 2024]
b. We developed a method that uses efficient GPU-based physical simulation to perform inference for robotic manipulation tasks, establishing the validity of using physics simulation to speed up Bayesian inference in robotics [Krupnik et al., Fine-tuning generative models as an inference method for robotic tasks. CoRL 2023].
c. Scaling up of deep Bayesian RL: we scaled up meta RL to domains with image inputs, showing that by exploring at test time we are able to improve the generalization to new domains in RL [Zisselman et al., Explore to generalize in zero-shot RL. NeurIPS 2023]. We also scaled up meta RL to higher dimensional task distributions, based on a novel model-based meta RL approach [Rimon et al., Mamba: an effective world model approach for meta-reinforcement learning. ICLR 2024]
Objective 3: Explore Safety Certificates for Deep RL
Following the research plan, work on this objective will start during year 3 of the project.
Objective 4: Develop a Practical Deep Learning Framework for Robotic Manipulation
a. We started investigating the problem of using simulation to speed up Bayesian inference in robotics [Krupnik et al., Fine-tuning generative models as an inference method for robotic tasks. CoRL 2023]
b. We have begun investigating a unified framework for robotic manipulation. We have made progress in deformable object manipulation (ropes) [Sudry et al., Hierarchical planning for rope manipulation using knot theory and a learned inverse model. CoRL 2023] and multi-object manipulation [Haramati et al., Entity-centric reinforcement learning for object manipulation from pixels. ICLR, 2024].
So far, the achievements above yielded 15 publications in top-tier machine learning conferences and journals.
