让强化学习策略学会复杂多模式决策，提升少样本鲁棒性。
Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
- 用重参数化技巧直接优化不可导的多模态策略
- 提出距离正则化提升策略多样性，无需显式概率计算
- 在多目标与生成式强化学习中表现优异，适合复杂决策场景
传统连续强化学习算法使用确定性或单模高斯策略，难以表达复杂的多模态决策分布，限制了其在需要多样性的场景中的表现。已有方法尝试基于扩散或代理化策略设计在线多模态强化学习算法，但这些策略不可导，导致现有方法难以同时兼顾性能、决策多样性和效率。为此，本文首次将现有不可导多模态策略统一纳入一个框架，并证明可通过重参数化进行策略梯度直接优化。进一步提出一种基于距离的多样性正则化，无需显式计算决策概率。在多目标达成和生成式强化学习两个关键领域验证了多模态策略的优势，尤其在少样本鲁棒性方面表现突出。在标准MuJoCo基准上，本方法也展现出竞争性性能。实验表明，代理化策略是一类兼具强多模态表达能力和高性能的有前景的策略模型。代码已开源：https://github.com/PneuC/DrAC
原文摘要 · Abstract (English)
Traditional continuous deep reinforcement learning (RL) algorithms employ deterministic or unimodal Gaussian actors, which cannot express complex multimodal decision distributions. This limitation can hinder their performance in diversity-critical scenarios. There have been some attempts to design online multimodal RL algorithms based on diffusion or amortized actors. However, these actors are intractable, making existing methods struggle with balancing performance, decision diversity, and efficiency simultaneously. To overcome this challenge, we first reformulate existing intractable multimodal actors within a unified framework, and prove that they can be directly optimized by policy gradient via reparameterization. Then, we propose a distance-based diversity regularization that does not explicitly require decision probabilities. We identify two diversity-critical domains, namely multi-goal achieving and generative RL, to demonstrate the advantages of multimodal policies and our method, particularly in terms of few-shot robustness. In conventional MuJoCo benchmarks, our algorithm also shows competitive performance. Moreover, our experiments highlight that the amortized actor is a promising policy model class with strong multimodal expressivity and high performance. Our code is available at https://github.com/PneuC/DrAC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。