TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM
Researchers have introduced TAD, a novel Temporal-Aware trajectory self-Distillation framework designed to enhance Diffusion Large Language Models (dLLMs). While dLLMs enable parallel text generation, they typically suffer from a trade-off where increasing tokens per forward pass reduces accuracy. TAD addresses this by conditioning a teacher model on prompts and ground-truth responses to generate decoding trajectories, recording intermediate masked states. The method partitions masked positions into near and distant subsets based remaining decoding steps. Near tokens are trained using hard cross-entropy loss for confident predictions, while distant tokens utilize soft KL divergence loss to preserve future planning knowledge. This approach allows for two deployment configurations: a Quality model prioritizing accuracy and a Speed model favoring acceleration. Experimental results on the LLaDA benchmark demonstrate significant improvements, raising average accuracy from 46.2% to 51.6% with the Quality model and increasing average AUP from 46.2 to 257.1 with the Speed model. The findings suggest TAD effectively balances speed and precision in generative AI, offering a robust solution for efficient parallel text generation without compromising output quality. The associated code has been made publicly available.
Wire timeline
TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM
Researchers have introduced TAD, a novel Temporal-Aware trajectory self-Distillation framework designed to enhance Diffusion Large Language Models (dLLMs). While dLLMs enable parallel text generation, they typically suffer from a trade-off where increasing tokens per forward pass reduces accuracy. TAD addresses this by conditioning a teacher model on prompts and ground-truth responses to generate decoding trajectories, recording intermediate masked states. The method partitions masked positions into near and distant subsets based remaining decoding steps. Near tokens are trained using hard cross-entropy loss for confident predictions, while distant tokens utilize soft KL divergence loss to preserve future planning knowledge. This approach allows for two deployment configurations: a Quality model prioritizing accuracy and a Speed model favoring acceleration. Experimental results on the LLaDA benchmark demonstrate significant improvements, raising average accuracy from 46.2% to 51.6% with the Quality model and increasing average AUP from 46.2 to 257.1 with the Speed model. The findings suggest TAD effectively balances speed and precision in generative AI, offering a robust solution for efficient parallel text generation without compromising output quality. The associated code has been made publicly available.
cs.AI updates on arXiv.org