通过模拟反向传播计算精确策略梯度，大幅减少采样需求。
Backpropagating Through Simulation: Analytic Policy Gradients for Sample and Learning Efficient Differentiable Continuous Control
- 将环境动力学建模为可微函数，用反向传播直接求梯度。
- 在4个连续控制任务中，仅需数千步即达与PPO相当的性能。
- 适合追求高效训练、有可微环境模型的研究者使用。
无模型强化学习算法如近端策略优化（PPO）将环境视为黑箱，通过采样奖励估计策略梯度，需数百万次交互且依赖高方差优势估计。当环境动力学可微时，回报可作为策略参数的端到端可微函数，实现通过模拟反向传播的精确梯度计算。本文提出分析型策略梯度（APG），并在四个动态复杂度递增的连续控制任务上与PPO对比：一维质点目标到达、二维带障碍物导航、二维刚体T块推移，以及7自由度Franka FR3末端执行器抓取任务。两算法使用相同模型架构、观测归一化和优化器设置。为分离采样效率与计算效率，设计多轴评估协议，记录性能随环境步数和梯度步数的变化。提出分段反向传播方案，包含蒙特卡洛与基于评价值的自举模式，有效缓解长时序任务中的梯度退化问题，并对分段长度与自举策略进行消融实验。
原文摘要 · Abstract (English)
Model-free reinforcement learning algorithms such as Proximal Policy Optimization (PPO) treat the environment as a black box, estimating policy gradients from sampled rewards; this process demands millions of interactions and relies on high-variance advantage estimates. When environment dynamics are differentiable, the return is an end-to-end differentiable function of the policy parameters, enabling exact gradient computation via backpropagation through simulation. We term this approach Analytic Policy Gradients (APG) and evaluate it against PPO on four continuous control tasks of increasing dynamical complexity: a one-dimensional point-mass target-reaching task, a 2D point-mass navigation task with obstacle avoidance, a 2D rigid-body T-block pushing task, and a 7-DOF Franka FR3 end-effector reaching task. Both algorithms share identical model architectures, observation normalization, and optimizer settings. To decouple sample efficiency from compute efficiency, we design a multi-axis evaluation protocol that records performance against environment steps and gradient steps. We report a segmented backpropagation scheme with MC and critic-based bootstrap modes that mitigates gradient degradation on long-horizon tasks, and present ablations over segment length and bootstrap strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。