Semantic Voting: Execution-Grounded Consensus for LLM Code Generation
A new research paper titled "Semantic Voting: Execution-Grounded Consensus for LLM Code Generation" has been published on arXiv, addressing the challenge of selecting optimal code outputs from Large Language Models (LLMs) without complete oracle access. The study evaluates 18 configurations across various models and benchmarks, comparing methods like output-pattern majority voting, weighted voting, MBR-Exec, and the proposed SemanticVote. Key findings indicate that execution-based selectors significantly outperform textual majority voting by 19-52 percentage points. Furthermore, once candidates are executed on diverse inputs, the specific aggregation rule has limited impact, with input quality emerging as the critical factor; sketch-based input generation proves superior to direct LLM generation or random fuzzing. The research also notes that while deeper thinking levels improve majority voting, they do not benefit execution-based methods due to reduced candidate diversity. Ultimately, the authors frame inference-time code selection as a signal-quality problem rather than an aggregation-rule issue, emphasizing that behavioral evidence is paramount when oracles are unavailable.
Wire timeline
Semantic Voting: Execution-Grounded Consensus for LLM Code Generation
A new research paper titled "Semantic Voting: Execution-Grounded Consensus for LLM Code Generation" has been published on arXiv, addressing the challenge of selecting optimal code outputs from Large Language Models (LLMs) without complete oracle access. The study evaluates 18 configurations across various models and benchmarks, comparing methods like output-pattern majority voting, weighted voting, MBR-Exec, and the proposed SemanticVote. Key findings indicate that execution-based selectors significantly outperform textual majority voting by 19-52 percentage points. Furthermore, once candidates are executed on diverse inputs, the specific aggregation rule has limited impact, with input quality emerging as the critical factor; sketch-based input generation proves superior to direct LLM generation or random fuzzing. The research also notes that while deeper thinking levels improve majority voting, they do not benefit execution-based methods due to reduced candidate diversity. Ultimately, the authors frame inference-time code selection as a signal-quality problem rather than an aggregation-rule issue, emphasizing that behavioral evidence is paramount when oracles are unavailable.
cs.AI updates on arXiv.org