Reinforcement Learning for Adaptive Parameter Control in Vascular Morphogenesis

Abstract

Controlling the morphology of vascular networks grown in silico involves solving an inverse problem: given a target tissue structure, determine the biophysical parameters needed to produce it. The Cellular Potts Model (CPM) is widely used to simulate vasculogenesis, but its high computational cost makes direct parameter optimization largely infeasible. In this thesis, we introduce a reinforcement learning (RL) framework that couples a pretrained U-Net neural surrogate of CPM dynamics with a Proximal Policy Optimization (PPO) agent to control two key CPM parameters dynamically---the cell-medium contact energy J and the VEGF decay constant k---over time with the goal of steering mean lacuna size toward a specified target. The evaluation protocol involves comparing three controllers: a fixed-parameter baseline, a static inverse baseline derived from a pre-computed parameter sweep, and the PPO agent trained in the surrogate environment. The PPO controller achieves a mean episode return of −1.70±1.68, which represents a 5.8× improvement over the static inverse baseline (−9.86±6.86), and reaches a steady-state tracking error of approximately 200 pixels compared to 700 pixels for the static method. Furthermore, the PPO agent converges to its target within approximately 20 surrogate steps, whereas the static baseline does not converge even after 100 steps. Analyzing the trajectory reveals that PPO can independently discover a two-phase control strategy: an initially aggressive organization phase with low J and low k, followed by a stabilization phase with high J and moderate k. Static parameter selection methods cannot discover this strategy, so these findings demonstrate that dynamic, state-conditioned parameter control poses a significant advantage over static approaches for guiding vascular morphogenesis in surrogate-based simulation environments.