Study Reveals Gradient Attribution Failures in Transformer Interpretability

A new research paper titled "Hidden Heroes and Gradient Bloats: Layer-Wise Redundancy Inverts Attribution in Transformers" challenges the reliability of gradient-based attribution in mechanistic interpretability. The study, authored by Donald Ye and published on arXiv, systematically evaluates whether gradient methods accurately track causal importance at the component level. Through experiments on algorithmic tasks like sequence reversal and sorting, the researchers uncover a significant layer-wise failure. They identify "Gradient Bloats" in early layers, which are overvalued by gradient attribution despite having negligible functional impact, and "Hidden Heroes" in late layers, which are undervalued but critical for performance. The rank correlation between gradient attribution and actual causal importance drops significantly, reaching negative values in some seeds. This discrepancy arises because first-order gradients fail to detect collective redundancy; joint ablation of Bloats causes far more damage than predicted. Conversely, removing Hidden Heroes severely degrades out-of-distribution accuracy. The findings suggest that current interpretability methods may be misleading and emphasize the need for causal validation before making circuit-level claims about transformer models.