arXiv:2606.21136cs.LGcs.RO2026-06

动态调整预测时长,让强化学习更准地规划长期任务。

Horizon Adaptive Offline Policy Learning via Value Stitching

论文配图:Horizon Adaptive Offline Policy Learning via Value Stitching
图 1 · 摘自论文原文
  • 用可变时长的值函数拼接替代固定时长更新。
  • 在OGBench上50个任务中表现优于现有方法,尤其擅长复杂长任务。
  • 适合需要精准长期规划的强化学习场景,如机器人决策。

在长时程、复杂任务中,准确的价值函数对强化学习代理至关重要。传统时序差分(TD)方法存在价值估计偏差随时间累积的问题;而扩展时程方法如n步TD和Q-chunking采用固定时程的建模方式,难以适应多阶段任务中的复杂价值结构。本文提出一种基于价值拼接的动态时程自适应离线策略学习方法(VAST),将固定时程更新替换为递归的、可自适应时长的价值组合机制。其核心是通过数据直接监督学习一个依赖未来状态与时长的辅助价值函数,并结合一个最优选择奖励最大时长和子目标的拼接策略,实现基于可行动子目标的可变长度回报的直接估计与组合。该设计为离线策略优化提供了准确且可贪婪利用的价值监督信号。在OGBench的50个任务上,VAST显著优于固定步长、扩展时程方法及生成式价值离线强化学习基线,在高复杂度、长时程决策任务中表现尤为突出。

原文摘要 · Abstract (English)

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.

强化学习离线学习长程规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。