解决强化学习连续控制中的动作抖动问题，部署时无延迟且无需后处理。
ZAPS-DA: Zero-Phase Action Policy Smoothing with Decoupled Actor for Continuous Control in Reinforcement Learning
- 用独立解耦的演员模仿缓冲区中零相位滤波目标，实现平滑动作输出。
- 在两个模拟器中将转向抖动降低14-21倍，油门抖动降低3-5倍，奖励损失仅6.3%。
- 无需调参即可跨优化器通用，对已训练策略可直接应用，适合工业部署。
基于离线策略强化学习训练的连续控制策略常出现高频动作抖动，难以直接用于物理执行器。事后滤波虽能抑制抖动但引入相位滞后；在演员损失中嵌入平滑惩罚会与强化学习梯度耦合，混淆奖励拟合与过度平滑。本文提出ZAPS-DA，通过一个未修改的主演员（以基础强化学习损失训练）与一个独立解耦的演员协作，后者通过监督学习模仿存储在回放缓冲区中的零相位滤波目标。部署时使用解耦演员：从观测到平滑动作的前馈映射，无推理时滤波，也无动作历史输入——即非因果滤波的因果蒸馏。采用幅度匹配的均方误差损失，实现跨优化器的零超参数可移植性。在软演员-评论家（SAC）和Savitzky-Golay滤波器下，于两个驾驶模拟器（配对样本量n=150）验证：在MetaDrive（基准协议）上，转向抖动降低14-21倍，油门抖动降低3-5倍（所有p<10⁻⁴，Bonferroni校正），任务完成率持平（p=0.28成功，p=0.31碰撞），奖励成本仅6.3%；在自定义Webots自适应巡航控制任务中，达到帕累托改进——奖励持平（p=0.121），转向抖动降低8-45倍，任务失败率由2.0%降至0.7%。相较标准惩罚基线CAPS（在自动熵与固定熵两种操作点，经环境级重调罚权重、空间噪声和熵系数），在相同种子下，ZAPS-DA实现14.7倍转向抖动降低，优于CAPS最佳3.2倍，差距约4.6倍，且无需为不同环境调整平滑信号，可直接应用于已训练策略。
原文摘要 · Abstract (English)
Continuous control policies trained with off-policy reinforcement learning frequently exhibit high-frequency action jitter, impractical for direct deployment on physical actuators. Post-hoc filtering attenuates jitter but adds phase lag; embedding smoothness penalties in the actor's loss couples them with the RL gradient and conflates reward regression with over-aggressive smoothing. We present ZAPS-DA, which reduces action jitter at deployment with negligible phase lag and no post-processing. ZAPS-DA pairs an unmodified main actor (trained by the base RL loss) with a separate decoupled actor trained via supervised imitation of zero-phase filtered targets stored in the replay buffer. The deployed policy is the decoupled actor: a feed-forward map from observation to smooth action, with no inference-time filter and no action-history input -- causal distillation of a non-causal filter. A magnitude-matched MSE loss gives zero-hyperparameter portability across optimizer classes. Validated with Soft Actor-Critic and a Savitzky-Golay filter in two driving simulators (paired n=150): on MetaDrive (anchor protocol), ZAPS-DA cuts steering jitter 14-21x and throttle jitter 3-5x (all $p<10^{-4}$, Bonferroni) while matching task-completion (p=0.28 success, p=0.31 crash) at 6.3% reward cost; on a custom Webots adaptive cruise control task, the same configuration yields a Pareto improvement -- reward parity (p=0.121), 8-45x steering-jitter reduction, task-failure rate 2.0% to 0.7%. Against CAPS, the standard penalty-based baseline -- at both its auto-entropy and native fixed-entropy operating points, with penalty weight, spatial noise, and entropy coefficient re-tuned per environment -- ZAPS-DA reaches 14.7x steering-jitter reduction versus CAPS's best 3.2x at matched seeds, a ~4.6x gap, with no per-environment tuning of the smoothness signal and post-hoc applicability to trained policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。