In-Context Linear Regression with Anisotropic Covariates

Abstract

We study how anisotropic covariance structure affects in-context learning (ICL) in a self-attention model for in-context linear regression. Existing asymptotic theory has been developed primarily in the isotropic setting, where rotational symmetry collapses the analysis to a small number of scalar order parameters. The goal of this thesis is to extend that theory to a structured anisotropic setting in order to determine which predictions of isotropic theory remain robust and which new phenomena emerge once directional structure is introduced. To do this, we introduce a split-spectrum Gaussian covariance model and derive an anisotropic extension of the isotropic framework. We develop a task-conditioned self-consistent theory and then take limits to remove task conditioning and study ICL in an asymptotic regime. Additionally, we use numerical simulations to evaluate and interpret the resulting theory curves. With the derived asymptotic theory, we find that several qualitative features of the isotropic theory persist; however, our theory also reveals novel phenomena. Most notably, we find that anisotropy can generate additional pre-interpolation peaks, yielding triple and quadruple descent patterns. These findings show that isotropic theory is informative but incomplete, and that structured covariance can qualitatively reshape ICL performance. The main contribution of this thesis is the derivation and validation of a finite-bank and asymptotic anisotropic extension that makes these effects analytically explicit.