Transformers Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

A new research paper published on arXiv demonstrates that standard transformers with softmax attention can effectively implement preconditioned Richardson iteration for in-context Gaussian kernel regression. While previous mechanistic accounts of in-context learning focused on linear tasks or functional-gradient dynamics for nonlinear cases, this study provides a convergent solver with end-to-end prediction-error guarantees. The authors construct a single-head transformer that approximates the kernel ridge regression predictor during its forward pass. Theoretical analysis reveals a functional decomposition where softmax attention handles cross-token interactions via row-normalized Gaussian-kernel operators, while ReLU MLP layers manage intra-token scalar arithmetic. Empirical experiments using GPT-2-style transformers trained on Gaussian-process regression tasks support these findings. Linear probing shows that the transformer's layer-wise predictions align closely with the step-wise outputs of classical solvers, specifically matching the error profiles of preconditioned Richardson iteration. This work identifies a concrete mechanism for nonlinear in-context learning, bridging the gap between transformer architecture and numerical optimization methods.