提出新框架CoDrift,让离线强化学习更稳定高效
CoDrift: Compositional Drifting for Offline Reinforcement Learning

- 用动作场统一建模多种目标,通过组合生成策略
- 在73个任务上表现优异,平均排名领先
- 适合做离线强化学习的算法研究者和工程师
离线强化学习本质上是多目标问题:策略需与固定数据集的行为支持保持一致,同时偏好高价值动作。本文将两类目标统一为动作空间中的运动场,使不同目标可直接通过场组合融合。受漂移模型启发,提出CoDrift——一种一步生成式策略学习的组合框架。该框架整合三种目标级场:条件场保留状态相关的行为结构,边际场在跨状态中聚合动作,提升连续控制离线强化学习中单正样本场景下的生成稳定性;价值场引导生成动作向高价值区域移动。最终组合场被嵌入随机生成器,在部署时仅需一次前向传播即可输出动作。在OGBench和D4RL的73个任务上评估,涵盖离线及离线转在线两种设置,CoDrift优于当前主流方法,两种设置下均取得最佳平均排名。
原文摘要 · Abstract (English)
Offline reinforcement learning is intrinsically multi-objective: a policy must remain compatible with the behavioral support of a fixed dataset while preferentially selecting high-value actions. We recast these objectives in a common form by viewing each as an action-space motion field that specifies how generated actions should move. This perspective enables heterogeneous learning objectives to be combined directly through field composition. Inspired by drifting models, we propose CoDrift, a compositional framework for one-step generative policy learning. CoDrift combines three objective-level fields into a unified policy field. The conditional field preserves state-dependent behavioral structure, while the marginal field pools actions across states to provide a more stable generative signal in the single-positive-sample regime of continuous-control offline RL. The value field moves generated actions toward higher-value regions. The composed field is absorbed into a stochastic generator that produces an action with a single forward pass at deployment. We evaluate CoDrift on 73 tasks from OGBench and D4RL in both offline and offline-to-online settings. CoDrift compares favorably with state-of-the-art methods and achieves the best average rank in both settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。