From Variational Principles to Differentiable Simulators: an Applied Exposition of Optimal Control

Abstract

This work examines three closely interrelated ideas: functional minimization, trajectory optimization, and optimal control. The term trajectory optimization typically describes offline path planning under known dynamics. On the other hand, optimal control typically denotes an optimization problem with feedback, requiring dynamic minimization of a time-dependent loss. Both problems can be understood as subsets of a broader class of functional minimization problems whereby a solution f∗ is sought to minimize a loss functional L : f → R. I cover a wide range of techniques in optimal control, ranging from classical ideas rooted in variational calculus to more modern approaches based on reinforcement learning and differentiable physics. My work revolves around two principal problems: the Brachistochrone, upon which I base much of my discussion of variational techniques, and an original control problem (the Rocket Problem) defined in Section 3. I implement three distinct computational solutions to the Rocket Problem: direct control policy optimization, a neural network controller, and a reinforcement learning agent trained via Proximal Policy Optimization. The first two approaches leverage a novel differentiable solver while the last takes a more generic approach to sequential decision problems.