MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
Researchers have introduced MURPHY, a novel multi-turn extension of Group Relative Policy Optimization (GRPO) designed to enhance self-correcting code generation in Large Language Models (LLMs). While standard GRPO methods are limited to single-turn interactions, MURPHY addresses the needs of agentic settings by constructing feedback-conditioned rollout trees. This approach pairs failed candidate solutions with executor feedback and expands them into subsequent turns, propagating rewards backward to credit earlier attempts that provided informative feedback. The study evaluates two propagation strategies, Max Reward (MARS) and Mean Reward (MERS), alongside post-rollout pruning mechanisms to optimize costs. Tested across three benchmarks (HumanEval, MBPP, LiveCodeBench-v6) and multiple model families including Qwen3 and OLMo-2, MURPHY achieved up to a 6% absolute improvement in pass@1 scores compared to prior multi-turn execution-feedback methods. Significant gains were observed in medium and hard difficulty subsets, highlighting the method's effectiveness in iterative self-correction tasks. This development represents a significant advancement in Reinforcement Learning with Verifiable Rewards (RLVR) for complex reasoning and coding applications.
Wire timeline
MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
Researchers have introduced MURPHY, a novel multi-turn extension of Group Relative Policy Optimization (GRPO) designed to enhance self-correcting code generation in Large Language Models (LLMs). While standard GRPO methods are limited to single-turn interactions, MURPHY addresses the needs of agentic settings by constructing feedback-conditioned rollout trees. This approach pairs failed candidate solutions with executor feedback and expands them into subsequent turns, propagating rewards backward to credit earlier attempts that provided informative feedback. The study evaluates two propagation strategies, Max Reward (MARS) and Mean Reward (MERS), alongside post-rollout pruning mechanisms to optimize costs. Tested across three benchmarks (HumanEval, MBPP, LiveCodeBench-v6) and multiple model families including Qwen3 and OLMo-2, MURPHY achieved up to a 6% absolute improvement in pass@1 scores compared to prior multi-turn execution-feedback methods. Significant gains were observed in medium and hard difficulty subsets, highlighting the method's effectiveness in iterative self-correction tasks. This development represents a significant advancement in Reinforcement Learning with Verifiable Rewards (RLVR) for complex reasoning and coding applications.
cs.AI updates on arXiv.org