提出一种轻量生成策略,实现高效高回报的连续控制。
Generative Actor-Critic with Soft Bridge Policies

- 用结构化桥接模型替代传统采样,仅需一次前向传播生成动作。
- 理论可解的相对熵目标,使策略优化更稳定且计算成本低。
- 适合对延迟敏感、追求高算力-收益比的强化学习应用。
表达性强的生成策略(如扩散模型和流模型)在最大熵在线强化学习中颇具吸引力,因其能建模多模态且高度非高斯的动作分布。然而,训练有效软生成策略面临两大挑战:一是边际动作密度通常不可得,现有方法依赖熵界、启发式代理或近似;二是迭代共享参数采样器导致推理开销大,需通过时间反向传播进行多次网络评估,增加内存占用并使策略优化不稳定。为此,本文提出软生成演员-评论家(SoftGAC),其演员在预-tanh空间中定义从固定基础潜在变量到终端动作潜在变量的随机桥接路径。该结构化桥接使最大熵目标转化为解析可解的路径级相对熵目标,相对于高熵参考过程。在有限步实现中,该相对熵精确退化为采样转移控制能量,从而提供原则性软正则化。同时,通过使用每步特定的小型桥接转移,每个动作仅需一次评估,保持演员轻量化,参数预算与强基线相当。大量实验表明,SoftGAC在具有挑战性的连续控制基准上,获得高于或相当的回报,优于扩散和流匹配策略,同时维持单次前向传播的低延迟特性,并显著提升计算-回报权衡表现。
原文摘要 · Abstract (English)
Expressive generative policies such as diffusion and flow models are appealing for MaxEnt online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and destabilizing policy optimization. These obstacles motivate us to seek a generative policy that exposes a tractable MaxEnt objective while requiring only a single sampled actor forward pass for action generation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor defines a stochastic bridge from a fixed base latent to a terminal action latent in pre-tanh space. This structured bridge allows us to lift the MaxEnt objective as an analytically tractable path-wise relative-entropy objective against a high-entropy reference process. In practical finite-step implementation, this relative entropy reduces exactly to sampled transition control energy and thus provides principled soft regularization. Moreover, we keep the single-pass actor lightweight by using small step-specific bridge transitions, each evaluated only once per sampled action, while maintaining a parameter budget comparable to strong actor baselines. Extensive experiments on challenging continuous-control benchmarks show that SoftGAC attains higher or competitive returns than strong generative policy baselines, including diffusion and flow-matching policies, while staying in the low-latency regime of one-pass actors and showing considerable improvements in the compute-return tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。