让强化学习在线适应更稳定，通过动态调整样本约束提升性能
From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning
- 基于行为一致性动态放松样本约束，取代静态数据分类
- 在D4RL上显著提升微调稳定性与最终表现
- 可适配多种算法，仅需每样本行为对齐
离线到在线强化学习（O2O RL）面临保留离线保守性与适应在线反馈之间的核心挑战，尤其在分布偏移下，数据行为随微调过程演化，使数据来源成为不可靠的约束依据，导致目标与数据不匹配。为此，我们提出动态对齐松弛框架DARE，基于行为模型的行为一致性实现样本级约束松弛。据我们所知，DARE是首个通过后验诱导交换机制，以行为一致性为条件进行约束松弛的方法，突破了离线/在线数据的二元划分。DARE仅需每样本行为对齐，可灵活集成至多种离线算法，支持不同行为模型与微调目标。理论分析表明，基于行为的样本交换能一致提升离线型与在线型子集的区分度。在D4RL上的实验显示，DARE在保持微调稳定性的同时，优于多个强基线，取得更优最终性能。（代码公开于：https://github.com/lpzu/DARE）
原文摘要 · Abstract (English)
Offline-to-online reinforcement learning (O2O RL) faces a central challenge between retaining offline conservatism and adapting to online feedback under distribution shift. This challenge arises because data behavior evolves during fine-tuning, rendering data origin a misleading basis for constraint handling and thereby leading to objective-data mismatch. We therefore propose Dynamic Alignment for RElaxation (DARE), a distribution-aware framework for sample-level constraint relaxation based on the behavioral consistency with a behavior model. To our knowledge, DARE is the first to condition constraint relaxation on behavioral consistency via a posterior-induced exchange mechanism, moving beyond a binary offline/online data distinction. Importantly, DARE requires only per-sample behavioral alignment, enabling instantiation on top of many offline algorithms with flexible choices of behavior models and fine-tuning objectives. We provide a theoretical analysis showing that behavior-based sample exchange consistently improves the distinction between offline-like and online-like subsets. Experiments on D4RL demonstrate that DARE consistently improves fine-tuning stability and achieves superior final performance over strong offline-to-online baselines. (The code is publicly available at \url{https://github.com/lpzu/DARE}.)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。