SpectraLDS: Distilling Spectral Filters into Constant-Time Recurrent Models

Abstract

We introduce the first provable method for learning a symmetric linear dynamical system of arbitrarily high effective memory. This allows us to distill the convolutional layers in a leading hybrid state space model, FlashSTU, into O(1) linear dynamical systems, merging Transformer and RNN architectures in a manner suitable for scaling and with application to language modeling and other sequential processing tasks.