用流匹配实现高效最大熵强化学习，推理速度提升显著。
Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching
- 通过重要性重加权规避分区函数计算，简化策略优化
- 设计解耦熵估计器，有效纠正离散化偏差，提升探索效率
- 结合MeanFlow实现一步生成，兼顾表达力与推理速度
扩散策略表达能力强但推理延迟高。流匹配（FM）可实现一步生成，但将其融入最大熵强化学习（MaxEnt RL）面临挑战：最优策略是难以处理的能量分布，且需高效估计对数似然以平衡探索与利用，却存在严重离散化偏差。本文提出FLAME框架，从三方面突破：第一，推导出基于Q值重加权的流匹配目标，避免分区函数估计；第二，设计解耦熵估计器，严格校正偏差，提升探索效率并逼近最优最大熵策略；第三，集成MeanFlow形式，实现表达丰富且高效的一步控制。在MuJoCo上的实验表明，FLAME优于高斯基线，性能媲美多步扩散策略，但推理开销显著更低。代码已开源。
原文摘要 · Abstract (English)
Diffusion policies are expressive yet incur high inference latency. Flow Matching (FM) enables one-step generation, but integrating it into Maximum Entropy Reinforcement Learning (MaxEnt RL) is challenging: the optimal policy is an intractable energy-based distribution, and the efficient log-likelihood estimation required to balance exploration and exploitation suffers from severe discretization bias. We propose \textbf{F}low-based \textbf{L}og-likelihood-\textbf{A}ware \textbf{M}aximum \textbf{E}ntropy RL (\textbf{FLAME}), a principled framework that addresses these challenges. First, we derive a Q-Reweighted FM objective that bypasses partition function estimation via importance reweighting. Second, we design a decoupled entropy estimator that rigorously corrects bias, which enables efficient exploration and brings the policy closer to the optimal MaxEnt policy. Third, we integrate the MeanFlow formulation to achieve expressive and efficient one-step control. Empirical results on MuJoCo show that FLAME outperforms Gaussian baselines and matches multi-step diffusion policies with significantly lower inference cost. Code is available at https://github.com/lzqw/FLAME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。