Latent Diffusion Policies

Abstract

Although reinforcement learning algorithms are often formulated with applicability to general environment state information, the majority of algorithm designs focus on direct representations of the state consisting of manipulator joint positions, an- gles, velocities, etc. While they are straightforward and human-interpretable, these representations do not reflect the vast dimensionality of information that would be en- countered by an agent in the real world, and as a result, such approaches oversimplify tasks like robotic control. One existing technique for reducing the huge amount of data lies in generative modeling, which also provides a general framework for manipulating information and generating new samples that can be used in reinforcement learning for a wide variety of purposes. To that end, this thesis project presents two novel techniques that use diffusion, a generative modeling technique, to simultaneously encode state representa- tions and generate high-quality behavior in simulated robotic environments. The two approaches consist of model-based and model-free systems, which are each compared to corresponding state-of-the-art implementations on the task of offline reinforcement learning. The results from this work demonstrate that latent diffusion policies are capable of performing at nearly the same level as raw-state policies, and that the limitations brought on by reducing state representations can be counteracted by parameter tun- ing and model design. Additionally, the findings from this project provide a basis for future work in reducing higher-dimensional observations such as video, in more challenging, real-world tasks such as robotics.