SMAC让离线训练的智能体在线微调时性能不下降，实现平滑迁移。
SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer
- 通过正则化Q函数，使策略梯度与动作梯度对齐，避免性能低谷。
- 在6个D4RL任务中实现0性能下降的平滑在线迁移。
- 适合需要稳定从离线到在线迁移的强化学习应用。
现代离线强化学习方法能获得高性能的策略-价值网络，但用基于价值的方法在线微调时通常导致性能骤降。我们发现，离线最优解与在线最优解之间存在性能低谷，梯度优化会穿越这些低谷。为此，我们提出分数匹配的策略-价值网络（SMAC），在离线阶段通过正则化Q函数，强制策略得分与动作梯度保持一阶导数一致性，从而避免陷入低谷。实验表明，SMAC收敛到的离线最优解可通过一阶优化路径连接到更高的在线最优解，且路径上奖励单调上升。在6个D4RL任务中，SMAC实现了与Soft Actor-Critic和TD3的平滑迁移；在4个环境中，其后悔值比最佳基线降低34%-58%。
原文摘要 · Abstract (English)
Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide evidence consistent with the hypothesis that, in the loss landscape, offline maxima for prior algorithms and online maxima are separated by low-performance valleys that gradient-based fine-tuning traverses. Following this, we present Score Matched Actor-Critic (SMAC), an offline RL method designed to learn actor-critics that transition to online value-based RL algorithms with no drop in performance. SMAC avoids valleys between offline and online maxima by regularizing the Q-function during the offline phase to respect a first-order derivative equality between the score of the policy and action-gradient of the Q-function. We experimentally demonstrate that SMAC converges to offline maxima that are connected to better online maxima via paths with monotonically increasing reward found by first-order optimization. SMAC achieves smooth transfer to Soft Actor-Critic and TD3 in 6/6 D4RL tasks. In 4/6 environments, it reduces regret by 34-58% over the best baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。