用部分状态信息也能高效训练强化学习模型，关键在选对辅助信号。
Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- critic 可基于任意状态相关辅助信号，不需完整状态信息
- 选对信号后性能可媲美甚至超过全状态方法
- 提出两种信号有效性评估方法，支持训练前与训练后使用
对称强化学习利用训练时的特权信息来提升部分可观测环境下的学习效果。现有方法通常假设训练时能访问完整环境状态以指导 critic，这在实践中往往不现实。本文提出知情不对称演员-评论家框架，允许 critic 基于任意依赖状态的特权信号进行条件化，并证明此类信号仍能提供无偏策略梯度估计。该设定显著拓展了可用特权信息的范围，也带来了如何选择最具信息量信号的问题。为此，我们提出两种新颖的信息性判别准则：一种是训练前可执行的依赖性测试，另一种是基于价值预测改进的训练后测试。在部分可观测基准和合成环境上的实验表明，经过精心挑选的特权信号可在仅依赖更少状态信息的情况下，实现与全状态不对称基线相当或更优的性能。
原文摘要 · Abstract (English)
Asymmetric reinforcement learning leverages privileged information available during training to improve learning under partial observability. Existing asymmetric actor-critic methods typically assume access to the full environment state to condition the critic during training, which is often unrealistic in practice. We introduce the informed asymmetric actor-critic framework that allows the critic to be conditioned on arbitrary state-dependent privileged signals, and show that any such signal yields unbiased policy gradient estimates. This substantially expands the set of admissible privileged information and raises the problem of selecting the most informative signals for learning. To this end, we propose two novel informativeness criteria: a dependence-based test that can be applied prior to training, and a test based on improvements in value prediction that can be applied post hoc. Experiments on partially observable benchmarks and synthetic environments demonstrate that carefully selected privileged signals can match or outperform full-state asymmetric baselines while relying on strictly less state information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。