Don't Explore What You Already Know: Pattern-seeking Exploration Via Temporal Invariance

Abstract

In reinforcement learning, the ability for an agent to succeed in a sparsely rewarding environment can hinge upon successful exploration. The efficiency of exploration is key for an agent to learn a successful policy optimally, and in certain environments, to succeed at all. This thesis seeks to create efficient exploration in environments with temporal invariance: that is, when there are dynamical patterns in the environment such that we can leverage the existence of a pattern and not re-explore re-instantiations of that pattern. Current methods define novelty in a manner such that they may not discover such patterns and thus repeatedly explore dynamically similar states. This paper introduces a method for exploration in these environments via pattern-seeking. Our method uses temporal contrastive learning, a learning method that draws consecutive states in time closer together in representation space, to produce a latent space that represents dynamically similar states very closely. We then form a covariance matrix of the representations of every state visited by the agent in an episode. We reward the agent for visiting states that are more novel when compared to its constructed covariance matrix. An agent trained with proximal policy optimization, a specific RL algorithm, is then trained with this intrinsic bonus. We measure the invariance of our representations and intrinsic bonus to spurious features and find that our method surpasses that of baseline methods. We also find that, when pre-training our representations, our method can solve environments with temporal invariance when baseline methods are incapable. Ultimately, this thesis presents a promising method to make RL policies pattern-seeking and paves the way for future work in pattern-seeking.