DARE: Efficient Inference for Diffusion Language Models via Activation Reuse
Researchers have introduced DARE, a novel optimization framework designed to enhance the efficiency of Diffusion Large Language Models (dLLMs). While dLLMs offer promising alternatives to auto-regressive models with potential for parallel generation, they currently lag in efficiency and quality. DARE addresses this by exploiting token-wise redundancy in bi-directional self-attention, where activations are highly correlated across tokens. The framework employs two complementary mechanisms: DARE-KV, which reuses cached key-value activations, and DARE-O, which reuses output activations. This approach significantly reduces redundant computation without requiring model retraining. Experimental results demonstrate that DARE achieves up to a 1.20x reduction in per-layer latency and reuses up to 87% of attention activations. Performance degradation is minimal, with average drops of only 2.0% for DARE-KV and 1.2% for DARE-O on reasoning and code-generation benchmarks. When combined with existing techniques like prefix caching, DARE provides additive efficiency gains. This development establishes token-wise reuse as an effective strategy for improving dLLM inference speed while maintaining generation fidelity, marking a significant step forward in making diffusion-based language models more practical for widespread application.
Wire timeline
DARE: Efficient Inference for Diffusion Language Models via Activation Reuse
Researchers have introduced DARE, a novel optimization framework designed to enhance the efficiency of Diffusion Large Language Models (dLLMs). While dLLMs offer promising alternatives to auto-regressive models with potential for parallel generation, they currently lag in efficiency and quality. DARE addresses this by exploiting token-wise redundancy in bi-directional self-attention, where activations are highly correlated across tokens. The framework employs two complementary mechanisms: DARE-KV, which reuses cached key-value activations, and DARE-O, which reuses output activations. This approach significantly reduces redundant computation without requiring model retraining. Experimental results demonstrate that DARE achieves up to a 1.20x reduction in per-layer latency and reuses up to 87% of attention activations. Performance degradation is minimal, with average drops of only 2.0% for DARE-KV and 1.2% for DARE-O on reasoning and code-generation benchmarks. When combined with existing techniques like prefix caching, DARE provides additive efficiency gains. This development establishes token-wise reuse as an effective strategy for improving dLLM inference speed while maintaining generation fidelity, marking a significant step forward in making diffusion-based language models more practical for widespread application.
cs.AI updates on arXiv.org