Learning from Imperfect Controllers: How Prior Structure Shapes Residual Reinforcement Learning

Abstract

Designing controllers that are both adaptable and interpretable remains a central challenge in reinforcement learning. Programmatic residual reinforcement learning aims to address this problem by learning corrections on top of a base program. The base program gives the system a starting point that a person can still inspect and understand, while leaving room for the residual to creatively solve remaining gaps in performance. This work investigates how using different base policies as programmatic priors influences learning behavior in continuous control tasks. Experiments are conducted in the Gymnasium Pendulum and LunarLander environments using three types of priors: a zero baseline, a hand-written controller, and controllers generated by a large language model (LLM). The expectation was that priors that encoded more knowledge would have stronger initial performance and therefore achieve optimal rewards after fewer training steps. However, the results show that initial performance alone does not determine learning success. Instead, the prior’s effectiveness depends on how well it aligns with the underlying system dynamics. In the Pendulum environment, many priors generated by the LLM exhibit an initial dip in performance due to the failure to apply torque in the direction of motion when trying to swing the pendulum toward the upright position. When this directional rule is explicitly included, LLM-generated policies achieve stable and efficient learning comparable to a hand-written expert. Similar trends are observed in the LunarLander environment, where residual learning is able to correct quantitative errors but fails when the prior is misaligned with the environment’s dynamics. These findings show that the effectiveness of residual reinforcement learning depends on the ability of the prior to generate trajectories that support learning, rather than simply achieving high initial reward. For LLM-generated policies, this highlights the importance of prompt design: language models can produce effective priors, but require explicit specification of key control principles. This work motivates future approaches for automatically generating and refining structurally correct programmatic priors.