轻量级演员搭配深度评论家,推理速度提升4倍。
Simple Actors and Deep Critics for Scalable Reinforcement Learning

- 用轻量确定性演员+深层评论家替代复杂生成式演员
- 在OGBench上性能媲美扩散模型,推理延迟降低4倍
- 适合对推理效率要求高的离线强化学习应用
近期离线强化学习进展依赖于扩散和流匹配等表达能力强的生成式演员,能捕捉离线数据集中的多模态行为。但这些演员每步动作需多次去噪或积分,部署时开销大。本文重新思考容量分配:由于评论家仅用于训练,而演员每步都运行,将容量投入评论家更利于推理效率。然而,深层评论家在离线RL中易引发优化、自举噪声放大和值范围漂移三类不稳定性。为此,提出残差MLP骨干、n步自举目标和分类交叉熵损失三个解决方案。结合轻量确定性演员,提出LAC(轻量演员,深层评论家)。在OGBench上,其性能与最强的扩散和流匹配基线相当,推理延迟降低最多4倍,接近单步蒸馏策略且无需蒸馏。该评论家设计可迁移至不同演员结构。
原文摘要 · Abstract (English)
Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。