Wire flash
Google: Gated modular edits outperform full rewrites in agent harness efficiency
Editorial responsibility
- No named human review is recorded for this page.
- Source reporting is collected, normalized, translated or condensed automatically when needed.
- Automatically published source-backed update
A new Google research paper reveals that self-improving agent harnesses achieve optimal results when utilizing gated modular edits rather than unchecked full rewrites. The study demonstrates that while a full rewrite of the agent harness matched accuracy levels, it consumed nearly ten times more tokens per task compared to the gated approach. The gated method involves an optimizer model improving the harness by editing one code module at a time, with each change required to pass validation on held-out tasks and adhere to a token cap. In testing on 40 SWE-bench Lite tasks using Gemini 3.8 Flash, the gated harness solved 40.0% of tasks using 46.7k tokens per task, whereas the unchecked full rewrite solved 42.5% but required 451.6k tokens. Additionally, relying solely on prompt rules resulted in a significant drop in accuracy to 2.5%. The findings suggest that enforcing changes through code-based gates and validation sets is superior to allowing agents to tune their own harnesses without constraints.
Source report
A new paper from Google reveals that while an unchecked full rewrite of an agent harness can match accuracy, it costs nearly 10 times more tokens per task compared to gated modular edits. The research concludes that self-improving agent harnesses perform best when they modify one code module at a time and strictly pass validation checks.
Understanding the Agent Harness
The "harness" refers to the surrounding code infrastructure that manages context, executes tools, and validates work before terminating. Key findings regarding these systems include:
- Portability Issues: A harness tuned for one specific model often stalls or wastes tokens when applied to another.
- Optimization Strategy: To address this, researchers allowed an optimizer model to improve a 142-line coding harness by editing only one of its four parts at a time.
- Validation Constraints: Every edit was subject to strict constraints; it had to pass a validation set and remain within a token cap, or it was automatically rejected.
Performance Comparison on SWE-bench Lite
Testing was conducted on 40 held-out SWE-bench Lite tasks using the Gemini 3.8 Flash model. The results highlight a significant trade-off between raw accuracy and efficiency:
| Approach | Accuracy | Tokens per Task | | :--- | :--- | :--- | | Gated Modular Edits (Proposed Method) | 40.0% | 46.7k | | Full Unchecked Rewrite | 42.5% | 451.6k | | Seed Model (Baseline) | 5.0% | N/A | | Prompt Rules Only | 2.5% | N/A |
While the full unchecked rewrite achieved a slightly higher accuracy rate (42.5%), it consumed 451.6k tokens, which is nearly ten times the cost of the gated approach. Conversely, relying solely on prompt rules resulted in a drastic drop in performance to just 2.5%.
Key Takeaways
To effectively tune agents without excessive resource consumption, the study recommends a three-pronged approach:
- Gate Changes: Validate every modification against held-out tasks.
- Enforce Cost Limits: Monitor and restrict token usage for each change.
- Code-Based Rules: Enforce logic and constraints directly in the code rather than relying on prompts.
Source
rohanpaul_aiNeutral / independent
Part of this Story
Google and Partners Reveal AI Self-Improvement Methods Boosting Code Performance