提出解耦策略与价值函数的行动块长度,提升长时序任务中的学习效率与性能。
Decoupled Q-Chunking
- 将策略与价值估计的行动块长度解耦,策略使用更短的块以增强反应性。
- 通过乐观回溯构建蒸馏价值函数,逼近部分动作块扩展后的最大值。
- 在长周期离线目标条件任务中显著优于现有方法,适合复杂动态环境。
时序差分(TD)方法通过自举未来价值预测高效学习状态和动作值,但易产生自举偏差,导致误差累积。近期工作引入分块批评者,对短动作序列(“块”)进行价值估计,加速价值传播。然而,从分块批评者中提取策略存在挑战:策略需开环输出整个动作块,这在需要实时反应的环境中表现不佳,且随块长增长建模难度上升。本文核心洞察是将批评者的块长与策略的块长解耦,使策略可基于更短的动作块运行。提出新算法,通过从原始分块批评者乐观回溯,构建用于部分动作块的蒸馏批评者,近似完整块扩展后的最大价值。该设计保留多步价值传播优势,同时避免开环子最优性和长块策略建模难题。在具有挑战性的长周期离线目标条件任务上验证,本方法稳定超越先前方法。代码见:github.com/ColinQiyangLi/dqc。
原文摘要 · Abstract (English)
Temporal-difference (TD) methods learn state and action values efficiently by bootstrapping from their own future value predictions, but such a self-bootstrapping mechanism is prone to bootstrapping bias, where the errors in the value targets accumulate across steps and result in biased value estimates. Recent work has proposed to use chunked critics, which estimate the value of short action sequences ("chunks") rather than individual actions, speeding up value backup. However, extracting policies from chunked critics is challenging: policies must output the entire action chunk open-loop, which can be sub-optimal for environments that require policy reactivity and also challenging to model especially when the chunk length grows. Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks. We propose a novel algorithm that achieves this by optimizing the policy against a distilled critic for partial action chunks, constructed by optimistically backing up from the original chunked critic to approximate the maximum value achievable when a partial action chunk is extended to a complete one. This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning action chunking policies for long action chunks. We evaluate our method on challenging, long-horizon offline goal-conditioned tasks and show that it reliably outperforms prior methods. Code: github.com/ColinQiyangLi/dqc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。