Wire flash
Northwestern and Google: AI Agents Self-Improve Without Answer Keys
Editorial responsibility
- No named human review is recorded for this page.
- Source reporting is collected, normalized, translated or condensed automatically when needed.
- Automatically published source-backed update
A new research paper from Northwestern University and Google introduces a method for AI agent harnesses to improve themselves on new tasks without relying on answer keys. An agent's harness comprises the prompts, tools, skills, and memory surrounding the model. The proposed system, Harness2, operates by running the agent, using an editor model to propose fixes, and then rerunning the task to verify which edits improve behavior. Successful lessons are stored in a library to guide future edits. In testing, a single improvement step raised GLM-5's WorkBuddy-Bench score by 11.5 points, surpassing Gemini 3.5 Flash on its stock harness. Conversely, editing the harness without these test runs caused coding scores to drop significantly from 73.4 to 63.2. The study concludes that treating the harness as a per-task revision target and retaining only edits that change real-world behavior is effective.
Source report
A recent collaborative paper from Northwestern University and Google introduces a novel method for agent self-improvement. The study demonstrates that agent harnesses can optimize their performance on new tasks without relying on answer keys, utilizing a process of test-running edits and retaining only those that yield positive results.
Understanding the Agent Harness
An agent's harness comprises the prompts, tools, skills, and memory structures wrapped around the underlying model. Traditionally, tuning these components by hand is a slow process. Furthermore, real-world tasks often lack the "answer keys" necessary to guide manual optimization.
How Harness2 Works
The proposed framework, Harness2, automates this optimization through a three-step iterative cycle:
- Execution: The system runs the agent on a specific task.
- Proposal: An editor model suggests potential fixes or improvements to the harness.
- Validation: The task is rerun to evaluate which specific fixes improved the agent's behavior.
Crucially, lessons learned from each task are stored in a centralized library. This repository then guides future edits for subsequent tasks, creating a cumulative knowledge base.
Key Performance Results
The efficacy of this approach was demonstrated through significant performance gains compared to traditional methods:
- Performance Boost: A single improvement step raised GLM-5's WorkBuddy-Bench score by 11.5 points.
- Competitive Edge: With its stock harness, GLM-5 surpassed Gemini 3.5 Flash following these optimizations.
- Risk of Manual Editing: Conversely, editing the harness without running these validation tests resulted in a sharp decline in coding scores, dropping from 73.4 to 63.2.
Conclusion
The research advocates for treating an agent's harness as a dynamic component that should be revised on a per-task basis. By strictly retaining only the edits that demonstrably change behavior during real-world runs, systems can achieve robust, autonomous improvement.
Source
rohanpaul_ai
Part of this Story
Google and Partners Reveal AI Self-Improvement Methods Boosting Code Performance