arXiv:2510.12638cs.LG2025-10被引 2

提出BWD方法,无需训练智能体即可评估离线强化学习数据质量。

Expert or not? assessing data quality in offline reinforcement learning

  • 用贝尔曼瓦舍尔斯坦距离衡量数据策略与随机策略的差异。
  • BWD在D4RL MuJoCo任务中与真实性能评分高度相关(皮尔逊系数0.92)。
  • 适合需要快速判断数据质量或优化策略时使用。

离线强化学习仅从静态数据集学习,不与环境交互。实际数据集质量参差不齐,常混合专家、次优甚至随机轨迹。算法选择依赖数据保真度:高质量数据可用行为克隆,混合或低质量数据则需能拼接跨轨迹有用行为的离线强化学习方法。但现实中难以预先判断数据质量,因数据来源和技能构成未知。本文研究从累积奖励到基于价值的估计器等各类代理指标,提出贝尔曼瓦舍尔斯坦距离(BWD),一种基于价值的最优传输评分,用于衡量数据行为策略与随机参考策略的差异。BWD通过行为批评家和状态条件最优传输公式计算,无需环境交互或完整策略优化。在D4RL MuJoCo任务中,BWD与聚合多个离线强化学习算法的基准性能评分强相关(皮尔逊相关系数0.92),可高效预测标准智能体在给定数据集上的表现。此外,将BWD作为正则项融入策略优化,能主动引导学习策略远离随机行为,提升回报。结果表明,如BWD这类价值感知的分布信号是实用的离线数据集筛选与策略优化工具。

原文摘要 · Abstract (English)

Offline reinforcement learning (RL) learns exclusively from static datasets, without further interaction with the environment. In practice, such datasets vary widely in quality, often mixing expert, suboptimal, and even random trajectories. The choice of algorithm therefore depends on dataset fidelity. Behavior cloning can suffice on high-quality data, whereas mixed- or low-quality data typically benefits from offline RL methods that stitch useful behavior across trajectories. Yet in the wild it is difficult to assess dataset quality a priori because the data's provenance and skill composition are unknown. We address the problem of estimating offline dataset quality without training an agent. We study a spectrum of proxies from simple cumulative rewards to learned value based estimators, and introduce the Bellman Wasserstein distance (BWD), a value aware optimal transport score that measures how dissimilar a dataset's behavioral policy is from a random reference policy. BWD is computed from a behavioral critic and a state conditional OT formulation, requiring no environment interaction or full policy optimization. Across D4RL MuJoCo tasks, BWD strongly correlates with an oracle performance score that aggregates multiple offline RL algorithms, enabling efficient prediction of how well standard agents will perform on a given dataset. Beyond prediction, integrating BWD as a regularizer during policy optimization explicitly pushes the learned policy away from random behavior and improves returns. These results indicate that value aware, distributional signals such as BWD are practical tools for triaging offline RL datasets and policy optimization.

离线RL数据质量最优传输策略评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。