Sketch-and-Verify: Structured Inference-Time Scaling via Program Sketching
Researchers from arXiv have introduced SKETCHVERIFY, a novel method for optimizing code generation in large language models (LLMs) under budget or latency constraints. This approach targets practitioners using smaller, cost-effective models like Gemini 3.1 Flash Lite. Instead of traditional flat sampling, SKETCHVERIFY factorizes the search space by having the LLM enumerate distinct algorithmic strategies and create program sketches with holes, which are then filled multiple times to generate structurally diverse candidates. These candidates are verified through execution and selected via fingerprint clustering. Experimental results on HumanEval+ demonstrate that within the same model tier, sketching significantly outperforms flat sampling in accuracy, recovering 58% of hard problems compared to 26% for flat sampling at matched costs. However, the study clarifies that this method does not replace upgrading to stronger model tiers, as greedy decoding on higher-tier models remains superior in both performance and cost-efficiency. The findings provide a clear operational rule: use sketching for cost-effective improvement within a tier, but upgrade tiers if possible. This research contributes to efficient AI deployment strategies in software engineering and machine learning.
Wire timeline
Sketch-and-Verify: Structured Inference-Time Scaling via Program Sketching
Researchers from arXiv have introduced SKETCHVERIFY, a novel method for optimizing code generation in large language models (LLMs) under budget or latency constraints. This approach targets practitioners using smaller, cost-effective models like Gemini 3.1 Flash Lite. Instead of traditional flat sampling, SKETCHVERIFY factorizes the search space by having the LLM enumerate distinct algorithmic strategies and create program sketches with holes, which are then filled multiple times to generate structurally diverse candidates. These candidates are verified through execution and selected via fingerprint clustering. Experimental results on HumanEval+ demonstrate that within the same model tier, sketching significantly outperforms flat sampling in accuracy, recovering 58% of hard problems compared to 26% for flat sampling at matched costs. However, the study clarifies that this method does not replace upgrading to stronger model tiers, as greedy decoding on higher-tier models remains superior in both performance and cost-efficiency. The findings provide a clear operational rule: use sketching for cost-effective improvement within a tier, but upgrade tiers if possible. This research contributes to efficient AI deployment strategies in software engineering and machine learning.
cs.AI updates on arXiv.org