动态调整动作块长度，让智能体更聪明地规划行为序列。
Adaptive Action Chunking via Multi-Chunk Q Value Estimation
- 用Transformer同时估算多种块长的收益，一次前向计算选出最优长度。
- 在34个挑战任务中表现优于固定块长方法，学习效率与泛化能力更强。
- 适合复杂环境下的强化学习任务，尤其看重行为连贯性的场景。
动作块化已成为模仿学习中的关键技术，使策略能预测连贯的动作序列而非单一动作。近期该方法扩展至强化学习（RL），提升了行为一致性并减少了价值函数估计中的自举误差。然而，现有方法依赖固定块长，导致性能瓶颈，因为最优块长随状态和任务变化。本文提出自适应动作块化（ACH）算法，一种从离线到在线的强化学习新方法，可在训练和推理阶段动态调节块长。为在动态变化的状态下找到最优块长，我们通过基于Transformer的架构，在单次前向传播中同时估算所有候选块长的行动价值。该机制使智能体能根据当前状态自适应选择最有效块长。在34个具有挑战性的任务上评估表明，ACH持续优于固定块长基线，展现出更强的泛化能力和学习效率。
原文摘要 · Abstract (English)
Action chunking emerged as a pivotal technique in imitation learning, enabling policies to predict cohesive action sequences rather than single actions. Recently, this approach has expanded to reinforcement learning (RL), enhancing behavioral consistency and reducing bootstrapping errors in value function estimation. However, existing methods rely on a fixed chunk length, creating a performance bottleneck as the optimal length varies across states and tasks. In this paper, we propose Adaptive Action CHunking (ACH), a novel offline-to-online RL algorithm that dynamically modulates chunk length during both training and inference. To find the optimal chunk length for a dynamically varying current state, we simultaneously estimate action-values for all candidate chunk lengths in a single forward pass, using a Transformer-based architecture. Our mechanism allows the agent to select the most effective chunk length adaptively based on the current state. Evaluated on 34 challenging tasks, ACH consistently outperforms fixed-length baselines, demonstrating superior generalization and learning efficiency in complex environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。