arXiv:2604.22873cs.LGcs.AI2026-04

冻结策略下用目标先验实现稳定部署调整,避免性能崩溃。

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

论文配图:When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning
图 1 · 摘自论文原文
  • 用专家乘积方法融合先验与冻结策略,保持安全稳定
  • 精度加权组合在先验劣化时仍保持原策略,优于其他方法
  • 适合无法重训练的工业场景,尤其需可靠部署的系统

离线强化学习可在固定数据集上训练有效策略,但部署目标可能变化。在无法重新训练模型(受数据、成本或治理限制)的情况下,我们研究了冻结策略的部署期适应问题。采用目标条件化的先验与专家乘积(PoE)组合,发现关键现象是性能渐进退化而非普遍提升:当先验质量下降或随机时,精度加权的PoE仍锚定于原策略,而加法和仅先验的调整方式会完全失效;同时,基于KL预算的选择器常能恢复接近最优的运行点。理论分析揭示,在对角高斯策略与先验条件下,带系数α的PoE等价于带β = α/(1−α)的KL正则化调整,后验协方差仅差一个全局缩放因子。在四个D4RL环境(共3,900个MuJoCo回合)中观察到4/5/3的HELP/FROZEN/HURT结果分布。扩展至六个更难任务及两个AntMaze诊断显示:中等专家水平策略在所有9个测试场景中始终为HURT,而行为克隆的冻结策略在AntMaze上对所有组合规则均零成功。总体而言,PoE与KL正则化适配应被视为统一的、以原策略为中心的部署期安全调控机制。

原文摘要 · Abstract (English)

Offline reinforcement learning (RL) can learn effective policies from fixed datasets, but deployment objectives may change after training, and in many applications the trained actor cannot be retrained because of data, cost, or governance constraints. We study deployment-time adaptation for frozen offline actors using Product-of-Experts (PoE) composition with a goal-conditioned prior. Our main practical finding is graceful degradation rather than universal performance gain: under degraded or random priors, precision-weighted composition remains anchored to the frozen actor, while additive and prior-only adaptation collapse, and a KL-budget selector often recovers a near-oracle operating point. We also make explicit a closed-form identity in the frozen-actor setting: for diagonal-Gaussian actors and priors, PoE with coefficient alpha yields the same deterministic policy as KL-regularized adaptation with beta = alpha / (1 - alpha), with posterior covariances differing only by a global scalar factor. Empirically, across four D4RL environments (3,900 MuJoCo episodes), we observe a 4/5/3 HELP/FROZEN/HURT split. Extending the analysis to six harder cells and two AntMaze diagnostics reveals an actor-competence ceiling: medium-expert remains HURT in all 9 cells at every tested alpha, while AntMaze with a behavior-cloned frozen actor yields zero success for all composition rules. Overall, PoE and KL-regularized adaptation are best viewed as a single actor-anchored safety mechanism for deployment-time steering.

强化学习离线学习部署适应策略冻结

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。