Step Rejection Fine-Tuning: A Practical Distillation Recipe for LLM Agents
Researchers have introduced Step Rejection Fine-Tuning (SRFT), a novel method for training Large Language Model (LLM) agents that improves upon the standard Rejection Fine-Tuning (RFT) approach. While traditional RFT discards entire unsuccessful trajectories, such as those failing software engineering tests in SWE-bench tasks, SRFT leverages these partially correct but unresolved paths. The new technique employs a critic LLM to evaluate the correctness of each individual step within a trajectory. During training, the model masks the loss for erroneous steps while retaining them in the context window, enabling the agent to learn error recovery without replicating mistakes. Evaluations on the SWE-bench Verified dataset demonstrate that SRFT increases the resolution rate by 3.7%, outperforming the 2.4% improvement achieved by standard RFT. This results in a total resolution rate of 32.2%. The study highlights a more efficient use of training data for complex tasks, suggesting that filtering rather than discarding failed attempts significantly enhances model performance in software engineering applications.
Wire timeline
Step Rejection Fine-Tuning: A Practical Distillation Recipe for LLM Agents
Researchers have introduced Step Rejection Fine-Tuning (SRFT), a novel method for training Large Language Model (LLM) agents that improves upon the standard Rejection Fine-Tuning (RFT) approach. While traditional RFT discards entire unsuccessful trajectories, such as those failing software engineering tests in SWE-bench tasks, SRFT leverages these partially correct but unresolved paths. The new technique employs a critic LLM to evaluate the correctness of each individual step within a trajectory. During training, the model masks the loss for erroneous steps while retaining them in the context window, enabling the agent to learn error recovery without replicating mistakes. Evaluations on the SWE-bench Verified dataset demonstrate that SRFT increases the resolution rate by 3.7%, outperforming the 2.4% improvement achieved by standard RFT. This results in a total resolution rate of 32.2%. The study highlights a more efficient use of training data for complex tasks, suggesting that filtering rather than discarding failed attempts significantly enhances model performance in software engineering applications.
cs.AI updates on arXiv.org