动态调整动作分块长度,让智能体在不同场景下更灵活决策。
Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

- 根据状态自适应选择最优动作分块大小,而非固定长度。
- 在OGBench和Robomimic上达到当前最佳的离线与在线成功率。
- 适合需要长程规划的复杂任务,如机器人操作和视觉语言动作模型。
离线到在线强化学习中,动作分块可消除多步偏离策略偏差并实现时间连贯探索,但现有方法对所有状态使用固定分块长度。这不理想:接近接触事件时需短分块以实现快速反应,而在自由空间运动时长分块更利于信用分配。自然解法是训练多个分块长度的评判器,并在每个状态下选择最优者,但直接比较学习到的评判值会因折扣尺度不匹配而系统性偏向最短分块,且在低价值状态退化为噪声。我们提出自适应Q分块(AQC),通过将每种分块长度的优势相对于每时段基线进行归一化对比,解决了上述问题。该准则在无真实信号时将错误答案转为无偏近随机选择,在存在有效信号时具备区分能力。我们证明了优势选择器的抗噪性理论边界,以及自适应分块优于任意固定分块的价值优势。实验表明,AQC在OGBench和Robomimic上达到最先进性能,并可提升大规模视觉-语言-动作模型在RoboCasa-GR1任务上的表现。
原文摘要 · Abstract (English)
Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive control, while during free-space motion long chunks provide better credit assignment. The natural solution is to train critics for several chunk sizes and select the best one at each state, but naive comparison of learned critic values systematically collapses to the shortest chunk due to discount-scale mismatch, and degrades to noise in low-value states. We propose Adaptive Q-Chunking (AQC), which resolves both failures by comparing the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor. This criterion converts biased wrong answers into unbiased near-random choices when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on the advantage selector's noise immunity and on the value dominance of adaptive chunking over any fixed chunk size. We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic, and can be applied to enhance the performance of large-scale VLA models that predict action sequences, significantly boosting performance on RoboCasa-GR1 tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。