Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination
A new research paper published on arXiv investigates the mechanistic causes of failure modes in large language models, specifically focusing on conflict and hallucination. The study reveals that both issues stem from the geometric properties of transformer memory. Parametric memory (facts in weights) and working memory (contextual information) interact within a hidden-state space where learned facts form attractor basins. Conflict occurs when working memory disrupts convergence to the correct basin, while hallucination happens when no such basin exists, causing the hidden state to drift. Crucially, the frozen output head cannot distinguish these states, leading to confident but incorrect outputs. The authors propose using 'geometric margin'—the distance to the nearest memorized basin—as a superior metric for detecting hallucinations compared to output entropy. This method effectively separates correct recall from errors without false refusals. The research confirms that this geometric structure is inherent to pretrained models and follows a specific scaling law, indicating that confident hallucinations may increase with model scale despite overall error reductions.
Wire timeline
Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination
A new research paper published on arXiv investigates the mechanistic causes of failure modes in large language models, specifically focusing on conflict and hallucination. The study reveals that both issues stem from the geometric properties of transformer memory. Parametric memory (facts in weights) and working memory (contextual information) interact within a hidden-state space where learned facts form attractor basins. Conflict occurs when working memory disrupts convergence to the correct basin, while hallucination happens when no such basin exists, causing the hidden state to drift. Crucially, the frozen output head cannot distinguish these states, leading to confident but incorrect outputs. The authors propose using 'geometric margin'—the distance to the nearest memorized basin—as a superior metric for detecting hallucinations compared to output entropy. This method effectively separates correct recall from errors without false refusals. The research confirms that this geometric structure is inherent to pretrained models and follows a specific scaling law, indicating that confident hallucinations may increase with model scale despite overall error reductions.
cs.AI updates on arXiv.org