提出新框架提升返回报条件监督学习性能，突破原方法上限。
How to Provably Improve Return Conditioned Supervised Learning?
- 引入分布内最优未来回报概念，指导策略优化
- 理论证明性能优于标准RCSL，实测多任务显著提升
- 适合想改进离线强化学习稳定性的研究者
在序列决策问题中，返回报条件监督学习（RCSL）因其简洁性和稳定性受到关注。与传统离线强化学习不同，RCSL将策略学习视为以状态和回报为输入的监督学习问题，避免了时序差分学习带来的不稳定性。然而，RCSL因缺乏拼接性质而受限，其性能受生成离线数据集的策略质量制约。为此，我们提出一种原理清晰、结构简单的框架——增强型RCSL。核心创新在于引入‘分布内最优回报-去’概念，利用当前策略识别基于当前状态的可实现最佳数据集内未来回报，无需复杂的回报增强技术。理论分析表明，增强型RCSL可持续优于标准RCSL。实证结果进一步验证了该方法在多个基准测试中的显著性能提升。
原文摘要 · Abstract (English)
In sequential decision-making problems, Return-Conditioned Supervised Learning (RCSL) has gained increasing recognition for its simplicity and stability in modern decision-making tasks. Unlike traditional offline reinforcement learning (RL) algorithms, RCSL frames policy learning as a supervised learning problem by taking both the state and return as input. This approach eliminates the instability often associated with temporal difference (TD) learning in offline RL. However, RCSL has been criticized for lacking the stitching property, meaning its performance is inherently limited by the quality of the policy used to generate the offline dataset. To address this limitation, we propose a principled and simple framework called Reinforced RCSL. The key innovation of our framework is the introduction of a concept we call the in-distribution optimal return-to-go. This mechanism leverages our policy to identify the best achievable in-dataset future return based on the current state, avoiding the need for complex return augmentation techniques. Our theoretical analysis demonstrates that Reinforced RCSL can consistently outperform the standard RCSL approach. Empirical results further validate our claims, showing significant performance improvements across a range of benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。