Continuous Latent Contexts Enable Efficient Online Learning in Transformers

A new research paper published on arXiv introduces a method for enhancing online learning capabilities in Transformer models through continuous latent contexts. While large language models excel at static in-context learning, they often struggle with adaptive, multi-turn online decision-making. This study demonstrates that constant-depth Transformers can implement foundational algorithms like the weighted majority algorithm and Q-learning by storing algorithmic states as linear combinations of feature embeddings within a small number of latent context tokens. The researchers trained a small GPT-2-style model using a multi-curriculum objective without direct supervision of latent states. Experimental results on long synthetic online prediction sequences show that this approach outperforms significantly larger and more complex models, including Qwen-3-14B and DeepSeek-V3. The findings suggest that continuous latent contexts offer a simple yet effective mechanism for maintaining persistent state in Transformers, enabling efficient online learning and adaptation in interactive environments without requiring parameter updates.