COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
Researchers have introduced COSAC, a novel critic-free per-agent policy gradient method designed for sequential cooperative teams where agents act in a fixed order and share a single team-level reward. Traditional critic-based approaches struggle with scalability as agent numbers increase, while existing critic-free alternatives suffer from issues like coupled noise or high variance. COSAC addresses these challenges by fitting an additive per-agent decomposition of the team reward using ridge regression, thereby decoupling learning signals from teammate noise. It computes counterfactual advantages through fictitious continuations of the current policy, eliminating the need for extra environment calls or importance-sampling reweighting. The method instantiates the Sequential Aristocrat Utility (SeqAU), extending previous utility concepts to sequential settings with proven bias and variance bounds. Empirical evaluations on sequential bandits and the AI2 Reasoning Challenge (ARC) demonstrate that COSAC achieves lower advantage mean squared error and faster convergence compared to other baselines, particularly in tasks involving multiple language models reasoning in turn.
Wire timeline
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
Researchers have introduced COSAC, a novel critic-free per-agent policy gradient method designed for sequential cooperative teams where agents act in a fixed order and share a single team-level reward. Traditional critic-based approaches struggle with scalability as agent numbers increase, while existing critic-free alternatives suffer from issues like coupled noise or high variance. COSAC addresses these challenges by fitting an additive per-agent decomposition of the team reward using ridge regression, thereby decoupling learning signals from teammate noise. It computes counterfactual advantages through fictitious continuations of the current policy, eliminating the need for extra environment calls or importance-sampling reweighting. The method instantiates the Sequential Aristocrat Utility (SeqAU), extending previous utility concepts to sequential settings with proven bias and variance bounds. Empirical evaluations on sequential bandits and the AI2 Reasoning Challenge (ARC) demonstrate that COSAC achieves lower advantage mean squared error and faster convergence compared to other baselines, particularly in tasks involving multiple language models reasoning in turn.
cs.AI updates on arXiv.org