让强化学习策略在训练时就考虑评估准确性，提升安全部署可靠性。
Evaluation-Aware Reinforcement Learning
- 训练时同步优化策略性能与可评估性，减少评估偏差和方差。
- 理论与实证结合证明可有效平衡评估精度与回报期望。
- 支持策略与评估机制联合学习，兼顾评估效果与性能表现。
策略评估是众多强化学习算法的核心组件，也是确保强化学习策略安全部署的关键工具。然而，现有策略评估方法常面临高方差或偏差问题。为此，我们提出评估感知强化学习（EvA-RL），一种通用的策略学习框架，在训练阶段即考虑评估准确性，而非传统的后验评估方式。具体而言，EvA-RL在优化策略性能的同时，直接追求高效且准确的评估能力。我们提供了一个EvA-RL的实例，并通过理论分析与实验结果表明，该方法能有效在评估准确性和期望回报之间实现权衡。最后，我们展示了评估感知策略与评估机制本身可协同学习，从而在不显著牺牲策略性能的前提下获得评估优势。本工作开辟了新研究方向，将可靠评估提升为强化学习中的首要原则。
原文摘要 · Abstract (English)
Policy evaluation is a core component of many reinforcement learning (RL) algorithms and a critical tool for ensuring safe deployment of RL policies. However, existing policy evaluation methods often suffer from high variance or bias. To address these issues, we introduce Evaluation-Aware Reinforcement Learning (EvA-RL), a general policy learning framework that considers evaluation accuracy at train-time, as opposed to standard post-hoc policy evaluation methods. Specifically, EvA-RL directly optimizes policies for efficient and accurate evaluation, in addition to being performant. We provide an instantiation of EvA-RL and demonstrate through a combination of theoretical analysis and empirical results that EvA-RL effectively trades off between evaluation accuracy and expected return. Finally, we show that the evaluation-aware policy and the evaluation mechanism itself can be co-learned to mitigate this tradeoff, providing the evaluation benefits without significantly sacrificing policy performance. This work opens a new line of research that elevates reliable evaluation to a first-class principle in reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。