Entropy-informed Decoding: Adaptive Information-Driven Branching

Researchers have introduced Entropy-informed Decoding (EDEN), a novel, model-agnostic framework designed to enhance the output quality of Large Language Models (LLMs). Current decoding strategies, such as sampling or fixed-width beam search, often struggle with either committing to single paths prematurely or expending excessive computational resources regardless of task complexity. EDEN addresses these limitations by adaptively allocating computation based on the model's uncertainty. It estimates the entropy of the output token distribution at each generation step, adjusting the branching factor monotonically. This approach expands more candidates in high-entropy regions while following greedier paths in low-entropy areas, thereby improving token efficiency. Theoretical proofs demonstrate that this method guarantees better continuations than fixed branching factors within the same budget. Experimental results across complex tasks, including mathematical reasoning, code generation, and scientific questioning, show that EDEN consistently outperforms existing strategies, offering superior accuracy-expansion trade-offs.