测试专家指导强化学习在不同情况下的可靠性，发现三类失败模式。
When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning
- 用统一基准对比多种专家引导方法，评估其在不同专家质量下的表现。
- 发现专家接近最优时，部分方法反而低于无专家的基线，且存在训练污染问题。
- 提出可量化的决策规则，帮助判断何时及如何信任专家指导。
许多连续控制任务自带一个性能良好但非最优的控制器（如调优的PID或手工设计步态）。越来越多的方法在强化学习中将此类控制器作为可查询的专家使用，但现有研究各自独立，基于不同基准且未评估不完美专家的影响。本文统一采用SAC主干、相同超参数优化与评估协议，每项任务-方法组合设置100/50个随机种子，并系统测试专家欠调优、动作偏差和观测噪声对性能的影响。结果揭示出三类被单一论文忽略的失败模式：(F1) 在argmax-plus-bootstrap机制下，价值函数出现盲点，导致IBRL在接近无专家强化学习上限的专家上表现反低于无专家的SAC；(F2) 在远离最优的专家上出现残余饱和；(F3) 热启动缓冲区污染，使训练时间交接方法在部署时专家欠调优下崩溃。无单一方法始终占优：各方法仅在特定任务结构下表现良好，其余则可预测地失效。在接近强化学习上限的专家（FourTank, GlassFurnace）上，所有查询时间方法均未能在100万步预算内超越专家，尚不清楚是根本限制还是预算不足所致。我们基于实验结果提炼出可测试的决策规则，依赖三个预训练可观测量（专家质量、任务终止条件、扰动类型）。该基准、分类体系与决策规则为本文主要贡献；此外还提出EDGE——一种基于软最大值集成置信上限的设计点，验证了分类体系所指出的两个可优化维度（门控形式、评分规则）均可单独利用。
原文摘要 · Abstract (English)
Many continuous-control problems ship with a competent but suboptimal controller (a tuned PID, a hand-designed gait). A growing family of methods uses such controllers as queryable experts during RL, but each method has been proposed in isolation, on a different benchmark, without imperfect-expert testing. We harmonize the comparison on a shared SAC backbone, common HPO and evaluation protocols, 100/50 seeds per (env, method), and a degradation sweep over expert undertuning, action bias, and observation noise. The comparison surfaces three failure modes single-paper evaluations miss: (F1) a critic blind spot under argmax-plus-bootstrap that drags IBRL below no-expert SAC on experts close to the no-expert-RL ceiling (RL-near-ceiling, distinct from the absolute physical ceiling); (F2) residual saturation on far-from-optimal experts; and (F3) warm-start buffer poisoning that collapses training-time-handoff methods under deployment-time expert undertuning. No single method dominates: each wins on one task-structure regime and fails predictably elsewhere; on RL-near-ceiling experts (FourTank, GlassFurnace) no query-time method clears the expert within our 1M-step budget, leaving open whether this is a fundamental wall or a budget effect. We convert the spread into a testable decision rule keyed on three pre-training observables (expert quality, task termination, perturbation type). The benchmark, taxonomy, and decision rule are the primary contribution; we additionally describe EDGE, a softmax-over-ensemble-LCB design point used to demonstrate that both axes the taxonomy points to (gate form, scoring rule) are individually exploitable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。