Adaptive, Satisficing Exploration

Abstract

In reinforcement learning, exploration is fundamental to how an agent learns to minimize regret. Most methods of exploration focus on the agent targeting an optimal policy in order to minimize regret. Other prior work explore the learning of satisficing policies instead. In this senior thesis, we look at the space in between, using satisficing policies to bridge towards an optimal policy. We build off the BLASTS algorithm, which generates β-satisficing policies, and explore adaptive functions of β based on the current time step and information encountered by the agent. From the respective episodic regrets, we show that time-based updates result in higher variance, while information-based updates stray from even satisficing policies. Analysis using controls on problem difficulty do provide some resolution that we can presume some ease of exploration for adapting β, as harder problems tend to be equally hard among different satisficing policies. We then build off the information-adaptive β to construct a curriculum-based approach towards updating β, providing one means of processing through the curriculum. We additionally show that temperature and β selection are key to understanding how the agent learns and how it performs, respectively. While this thesis does not produce a strictly more optimal algorithm than BLASTS, it stays competitive, which shows there is promise in using satisficing policies to learn optimal ones. It also lays groundwork for future work to analyze alternative schemes towards learning a curriculum of satisficing policies with the goal of optimality.