VeriContest: A New Benchmark for Verifiable Code Generation in AI
Researchers have introduced VeriContest, a comprehensive benchmark designed to evaluate the ability of large language models (LLMs) to generate verifiable code. Unlike traditional coding benchmarks that focus solely on executable output, VeriContest requires models to produce formal specifications and machine-checkable proofs alongside Rust code using the Verus verification system. The dataset comprises 946 competitive programming problems sourced from LeetCode and Codeforces, each validated through a rigorous three-phase pipeline involving expert review and automated testing. Evaluation of ten state-of-the-art AI models reveals a significant performance gap: while the strongest model achieved 92.18% accuracy in standard code generation, it only reached 5.29% in end-to-end verified program synthesis. These findings highlight specification and proof generation as critical bottlenecks in current AI systems. VeriContest aims to provide a robust platform for measuring and improving the correctness guarantees of AI-generated software, addressing the lack of reliable benchmarks in this emerging field of software engineering and artificial intelligence.
Wire timeline
VeriContest: A New Benchmark for Verifiable Code Generation in AI
Researchers have introduced VeriContest, a comprehensive benchmark designed to evaluate the ability of large language models (LLMs) to generate verifiable code. Unlike traditional coding benchmarks that focus solely on executable output, VeriContest requires models to produce formal specifications and machine-checkable proofs alongside Rust code using the Verus verification system. The dataset comprises 946 competitive programming problems sourced from LeetCode and Codeforces, each validated through a rigorous three-phase pipeline involving expert review and automated testing. Evaluation of ten state-of-the-art AI models reveals a significant performance gap: while the strongest model achieved 92.18% accuracy in standard code generation, it only reached 5.29% in end-to-end verified program synthesis. These findings highlight specification and proof generation as critical bottlenecks in current AI systems. VeriContest aims to provide a robust platform for measuring and improving the correctness guarantees of AI-generated software, addressing the lack of reliable benchmarks in this emerging field of software engineering and artificial intelligence.
cs.AI updates on arXiv.org