Emergence of Analogies through Geometric Structure Regularization in Self-Supervised Reinforcement Learning

Abstract

Solving real-world control problems requires generalizing across new variations without extensive, if any, retraining. While modern goal-conditioned reinforcement learning algorithms can learn representations to succeed at the task they were trained on, they fail to preserve relational structure between tasks and goals, often leading to limited ability to reuse learned skills and poor out-of-domain generalization. In this work, we aim to define a useful notion of analogies for control and investigate when analogies arise in control tasks, allowing us to design methods that can learn representations that enable robust analogical reasoning. We find that regularization on the geometric structure of representations (such as smaller representation dimensions, sparsity regularizers, and matching difference vectors of analogous pairs) can lead up to 3-18x increase in downstream task performance and moderate increases in out-of-distribution generalization while also inducing desirable analogical structures in the learned representations. Notably, we do not observe the same effects when compressing the information of the representations through information bottlenecks. We show a theoretical result that the contrastive RL (CRL) critic already learns weak action-conditioned analogies based on reachability and provide a lightweight method that uses the CRL critic to define analogous equivalences, eschewing the need for separate bisimulation metrics. We find that geometric structure regularization appears to be more critical in producing desirable representation structure than information bottlenecking alone. Rather than claiming that learned analogies cause performance gains, we argue that both structure and downstream performance emerge from a result of deeper structural alignment with the intrinsic task dimensionality. We believe this work is a first step toward understanding emergent analogical reasoning for control and designing intelligent decision-making systems that learn efficiently, reason about the environment through past experience, and, ultimately, act effectively in novel situations.