用分块引导提升离线强化学习的长期规划能力
Chunk-Guided Q-Learning
- 用分块值函数指导单步学习，平衡误差累积与精细控制
- 在长周期任务上性能超越传统单步与分块方法
- 适合需要长期决策的复杂强化学习场景
在离线强化学习中，单步时序差分（TD）学习因误差累积导致长期表现下降。虽然动作分块的TD方法通过多步回溯缓解此问题，但会限制策略为开环动作序列，引入次优性。为此，我们提出分块引导Q学习（CGQ），一种单步TD算法，通过将细粒度单步评判器正则化至基于扩展时间回溯训练的分块评判器，减少误差累积同时保留精细价值传播。理论上，CGQ优于单独使用单步或分块TD学习的最优性边界。实验表明，CGQ在具有挑战性的长周期OGBench任务上表现优异，通常超越单步和分块方法。
原文摘要 · Abstract (English)
In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。