Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation

This academic paper addresses critical limitations in current benchmark evaluation methods for Large Language Models (LLMs). Existing approaches often rely on deterministic strategies or single random samples, failing to account for the inherent randomness of LLMs. This oversight leads to unaccounted sampling variance and unreliable score estimates. The authors propose a hierarchical statistical model that incorporates both benchmark characteristics and LLM randomness to provide a more comprehensive assessment. The study demonstrates that leveraging multiple generations significantly improves the accuracy of benchmark score estimation and reduces variance. Furthermore, this approach enables the definition of a prompt-level difficulty score based on correct ratios, offering fine-grained insights into individual prompts. The researchers also introduce a data map visualizing prompt difficulty and semantics, which facilitates error detection and quality control in benchmark construction. Published on arXiv, this work aims to enhance the reliability and precision of LLM evaluations, providing a robust framework for assessing model capabilities in natural language processing tasks.