用Transformer让强化学习批评家看懂长序列，提升复杂任务表现
Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns
- 用轻量Transformer建模轨迹上下文，结合多步回报更新批评家
- 在长轨迹控制任务中超越标准SAC，最大提升达30%以上
- 无需目标网络和重要性采样，适合稳定训练长时序任务
我们提出一种序列条件化的批评家模型，用于软动作-评价（SAC）算法。该模型通过轻量级Transformer捕捉短轨迹片段的上下文信息，并基于聚合的N步目标进行训练。与以往方法不同，本方法不孤立评估状态-动作对，也不依赖演员端的动作分块来处理长时序问题，而是通过增强批评家自身对序列信息的感知能力，直接整合多步回报，无需重要性采样（IS）。所得到的序列感知价值估计能有效捕捉长期任务中的关键时间结构。在局部运动基准测试中，我们进一步发现冻结批评家参数若干步后，其更新方式与CrossQ的核心思想兼容，可在无目标网络的情况下实现稳定训练。尽管结构简单——仅使用两层隐藏单元数为128-256的Transformer，且最大更新到数据比（UTD）为1——该方法在多个任务上持续优于标准SAC及强基线方法，尤其在长轨迹控制任务中表现显著提升。这些结果凸显了在批评家侧引入序列建模与N步自举对长时序强化学习的重要价值。
原文摘要 · Abstract (English)
We introduce a sequence-conditioned critic for Soft Actor-Critic (SAC) that models trajectory context with a lightweight Transformer and trains on aggregated $N$-step targets. Unlike prior approaches that (i) score state-action pairs in isolation or (ii) rely on actor-side action chunking to handle long horizons, our method strengthens the critic itself by conditioning on short trajectory segments and integrating multi-step returns -- without importance sampling (IS). The resulting sequence-aware value estimates capture the critical temporal structure for extended-horizon and sparse-reward problems. On local-motion benchmarks, we further show that freezing critic parameters for several steps makes our update compatible with CrossQ's core idea, enabling stable training \emph{without} a target network. Despite its simplicity -- a 2-layer Transformer with 128-256 hidden units and a maximum update-to-data ratio (UTD) of $1$ -- the approach consistently outperforms standard SAC and strong off-policy baselines, with particularly large gains on long-trajectory control. These results highlight the value of sequence modeling and $N$-step bootstrapping on the critic side for long-horizon reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。