Attention Drift: What Autoregressive Speculative Decoding Models Learn
Researchers have identified a phenomenon termed 'attention drift' in speculative decoding models used to accelerate Large Language Model (LLM) inference. This issue causes drafter models to degrade significantly under template perturbation and long-context inputs, as attention shifts from the original prompt to recently generated tokens within the speculation chain. The study attributes this drift to un-normalized residual paths, where the drafter's hidden state magnitude grows monotonically with chain depth. To address this, the authors propose two architectural modifications: applying post-norm to drafter hidden states and implementing per-hidden-state RMSNorm after capturing target hidden states. These interventions significantly improve performance, doubling acceptance length under template perturbation and enhancing results on long-context tasks and standard benchmarks compared to the leading pre-norm EAGLE3 model. The findings suggest that these changes allow shorter training depths to generalize effectively over longer drafting sequences, offering a robust solution for improving the efficiency and reliability of autoregressive speculative decoding in AI systems.
Wire timeline
Attention Drift: What Autoregressive Speculative Decoding Models Learn
Researchers have identified a phenomenon termed 'attention drift' in speculative decoding models used to accelerate Large Language Model (LLM) inference. This issue causes drafter models to degrade significantly under template perturbation and long-context inputs, as attention shifts from the original prompt to recently generated tokens within the speculation chain. The study attributes this drift to un-normalized residual paths, where the drafter's hidden state magnitude grows monotonically with chain depth. To address this, the authors propose two architectural modifications: applying post-norm to drafter hidden states and implementing per-hidden-state RMSNorm after capturing target hidden states. These interventions significantly improve performance, doubling acceptance length under template perturbation and enhancing results on long-context tasks and standard benchmarks compared to the leading pre-norm EAGLE3 model. The findings suggest that these changes allow shorter training depths to generalize effectively over longer drafting sequences, offering a robust solution for improving the efficiency and reliability of autoregressive speculative decoding in AI systems.
cs.AI updates on arXiv.org