LanBo: Breaking the Overscaling Curse in LLM Parallel Thinking
Researchers from arXiv have published a study addressing the 'Overscaling Curse' in Large Language Model (LLM) reasoning, where system efficacy contradicts sample efficiency. While parallel thinking enhances reasoning through multi-path sampling, standard evaluations apply a global budget to all samples, leading to inefficient resource utilization as many samples require significantly smaller budgets for optimal accuracy. To resolve this, the authors propose the Latent Budget Predictor (LanBo), which analyzes model latent representations to predict sample-specific optimal budgets. This approach significantly improves budget utilization without compromising dataset accuracy. Furthermore, the study introduces Pre-decoding Budget Adaptation (PreAda), a paradigm integrating LanBo into the decoding pipeline to allocate budgets before decoding, thereby preserving parallelization. The results demonstrate that LanBo substantially enhances hardware-aware efficiency regarding latency and memory usage. This research offers a practical solution for optimizing parallel decoding in LLMs, highlighting the potential for more efficient AI systems by balancing computational costs with performance requirements through adaptive budgeting strategies.
Wire timeline
LanBo: Breaking the Overscaling Curse in LLM Parallel Thinking
Researchers from arXiv have published a study addressing the 'Overscaling Curse' in Large Language Model (LLM) reasoning, where system efficacy contradicts sample efficiency. While parallel thinking enhances reasoning through multi-path sampling, standard evaluations apply a global budget to all samples, leading to inefficient resource utilization as many samples require significantly smaller budgets for optimal accuracy. To resolve this, the authors propose the Latent Budget Predictor (LanBo), which analyzes model latent representations to predict sample-specific optimal budgets. This approach significantly improves budget utilization without compromising dataset accuracy. Furthermore, the study introduces Pre-decoding Budget Adaptation (PreAda), a paradigm integrating LanBo into the decoding pipeline to allocate budgets before decoding, thereby preserving parallelization. The results demonstrate that LanBo substantially enhances hardware-aware efficiency regarding latency and memory usage. This research offers a practical solution for optimizing parallel decoding in LLMs, highlighting the potential for more efficient AI systems by balancing computational costs with performance requirements through adaptive budgeting strategies.
cs.AI updates on arXiv.org