arXiv:2605.22376cs.LG2026-05

用目标域贝尔曼目标对齐筛选源域数据,提升跨域离线强化学习效果。

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning

论文配图:Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
图 1 · 摘自论文原文
  • 基于目标域贝尔曼目标评估源域数据转移性,而非表面相似性。
  • 在目标域数据极少时仍能稳定提升策略性能。
  • 适合数据稀缺的跨域强化学习场景,尤其适用于迁移学习需求者。

跨域离线强化学习(CDRL)旨在利用源域数据提升目标域的策略学习效果。现有方法通常通过衡量源域与目标域状态转移的相似性来评估数据可迁移性,并隐式进行过渡层面的选择:相似过渡被赋予更高权重或奖励,不相似者则被降权。然而,过渡层面的相似性并不必然意味着长期回报的一致性。即使视觉或动态上相似的过渡,在目标域中也可能导致显著不同的结果,从而误导策略学习并降低性能。为此,我们重新审视策略学习的根本目标:由于策略优化最终依赖于贝尔曼目标来评估决策质量,我们提出基于源域过渡与目标域贝尔曼目标的对齐程度来评估其可迁移性,而非表层相似性。基于此,我们提出一种名为目标对齐贝尔曼备份(Target-Aligned Bellman Backup, TABB)的方法,通过测量源域数据对目标域贝尔曼目标估计的贡献,选择性地利用源域数据。我们在多种跨域离线强化学习设置下进行了评估,目标域数据极度有限。实验结果表明,TABB始终表现优异,显著优于现有方法。

原文摘要 · Abstract (English)

Cross-domain offline reinforcement learning (CDRL) aims to improve policy learning in a target domain by leveraging data collected from a source domain. Existing works typically assess the transferability of source-domain data by measuring its similarity to target-domain transitions, and implicitly perform transition-level selection. Transitions that are considered similar are assigned higher weights or rewards, while dissimilar ones are down-weighted. However, transition-level similarity does not necessarily imply consistency in long-term returns. Even visually or dynamically similar transitions may lead to significantly different outcomes in the target domain, which can mislead policy learning and degrade performance. To address this issue, we revisit the fundamental objective of policy learning. Since policy optimization ultimately relies on Bellman targets to evaluate the quality of decisions, we propose to assess the transferability of source-domain transitions based on their alignment with target-domain Bellman targets, rather than superficial transition similarity. Based on this insight, we propose a method termed Target-Aligned Bellman Backup (TABB), which selectively leverages source-domain data by measuring their contribution to accurate Bellman target estimation in the target domain. We evaluate TABB across a broad range of cross-domain offline RL settings with highly limited target-domain data. Experimental results show that TABB consistently achieves strong performance.

强化学习跨域迁移离线学习贝尔曼备份

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。