Diagnosing Query-Time Expert-Guided Reinforcement Learning

Researchers from the artificial intelligence community have released a new study addressing the integration of suboptimal controllers, such as tuned PID systems, as queryable experts in Reinforcement Learning (RL). The paper harmonizes comparisons across various methods using a shared Soft Actor-Critic (SAC) backbone and rigorous evaluation protocols, including extensive seed testing and degradation sweeps. The analysis identifies three critical failure modes previously overlooked: critic blind spots near RL performance ceilings, residual saturation with poor experts, and buffer poisoning during training-time handoffs. The findings indicate that no single method dominates all scenarios; instead, performance varies by task structure. The authors propose a testable decision rule based on pre-training observables like expert quality and perturbation type. Additionally, they introduce EDGE, a new design point demonstrating the exploitability of gate forms and scoring rules. This work provides a comprehensive benchmark and taxonomy for improving expert-guided RL systems in continuous-control problems.