Spectral Filtering for Sample Efficient World Modeling in Reinforcement Learning

Abstract

World models have recently emerged as one of the dominant paradigms for model-based reinforcement learning. However, prediction errors in neural network based world models compound rapidly during autoregressive rollouts, limiting their utility for long-horizon planning. This thesis explores the use of spectral filtering via Spectral Transform Units (STUs) through various integrations into world model frameworks, primarily the EfficientZero V2 (EZV2) framework and the DreamerV3 framework. Experiments are conducted on 24 games in the Atari 100K benchmark and 4 tasks in the DeepMind Control Suite. In DreamerV3, integrating STU inside the recurrent state space model accelerates convergence by 36% on walker walk and achieves 66% higher asymptotic performance on walker run. Filter basis ablations confirm that structured spectral bases drive this improvement; random orthonormal filters converge slower than baseline. In EZV2, applying STU as a spatial mixer over the latent state grid outperforms baseline dynamics on all 24 tested Atari games at 5 training episodes, with step-50 rollout MSE advantages ranging from 1.1× to 65.6×. The advantage grows with prediction horizon, and stability gains hold with fewer parameters than baseline, suggesting that stability is an emergent property of the spectral architecture. However, spectral filtering outperforms attention on only 11 of the 24 environments tested. Together, these results indicate that spectral filtering may provide a useful inductive bias for world model dynamics, improving both rollout stability and online training efficiency across discrete and continuous control domains.