arXiv:2502.00601cs.LG2025-02中稿 · AAMAS 2025

通过课程学习评估轨迹,提升离线强化学习在跨域数据下的性能。

Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation

  • 基于转移相似性评分,识别目标域高质量轨迹。
  • 在多个MuJoCo环境上提升策略性能与迁移能力。
  • 适合处理源域数据多、目标域数据少的离线学习场景。

深度强化学习的成功依赖于充足且高质量的训练数据,通常需与特定环境进行大量交互。在数据采集成本高且存在风险的真实场景中,离线强化学习利用领域专家收集的数据,在批量约束下搜索最优策略。该方法还可结合外部数据源,拓展数据多样性。然而,现有离线强化学习方法常面临外部数据源与目标域不匹配的挑战。本文针对混合数据集中源域数据占主导(由随机或次优策略生成)而目标域数据有限(由高质量策略生成)的情况,提出转换评分(TS),根据转移样本与目标域的相似性分配得分,并设计课程学习式轨迹估值(CLTV),通过课程学习机制利用这些得分筛选并优先训练高质量轨迹。在多种离线强化学习方法和MuJoCo环境上的广泛实验,结合严格的理论分析表明,CLTV显著提升了离线强化学习算法所学策略的整体性能与可迁移性。

原文摘要 · Abstract (English)

The success of deep reinforcement learning (DRL) relies on the availability and quality of training data, often requiring extensive interactions with specific environments. In many real-world scenarios, where data collection is costly and risky, offline reinforcement learning (RL) offers a solution by utilizing data collected by domain experts and searching for a batch-constrained optimal policy. This approach is further augmented by incorporating external data sources, expanding the range and diversity of data collection possibilities. However, existing offline RL methods often struggle with challenges posed by non-matching data from these external sources. In this work, we specifically address the problem of source-target domain mismatch in scenarios involving mixed datasets, characterized by a predominance of source data generated from random or suboptimal policies and a limited amount of target data generated from higher-quality policies. To tackle this problem, we introduce Transition Scoring (TS), a novel method that assigns scores to transitions based on their similarity to the target domain, and propose Curriculum Learning-Based Trajectory Valuation (CLTV), which effectively leverages these transition scores to identify and prioritize high-quality trajectories through a curriculum learning approach. Our extensive experiments across various offline RL methods and MuJoCo environments, complemented by rigorous theoretical analysis, demonstrate that CLTV enhances the overall performance and transferability of policies learned by offline RL algorithms.

离线RL课程学习轨迹评估跨域迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。