聚焦单一行为模式，提升离线强化学习在多模态数据中的表现
Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning
- 从行为数据中识别最优模式，仅基于该模式进行模仿学习
- 在D4RL标准基准上超越现有方法，复杂场景下表现更优
- 适合处理多行为共存的离线强化学习任务
离线强化学习旨在不与环境交互的情况下，仅从静态数据集学习最优策略。常见挑战是处理多模态动作分布，即数据中存在多种行为模式。现有方法通常假设行为策略为单模态，当该假设不成立时性能下降。本文提出加权模仿学习在单一模式（LOM），专注于从一个有前景的行为模式中学习。通过高斯混合模型识别模式，并根据预期回报选择最佳模式，LOM避免了对冲突动作的平均化。理论上，我们证明了LOM能提升性能并保持策略学习的简洁性。实验表明，LOM在标准D4RL基准上优于现有方法，并在复杂多模态场景中表现出色。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without interacting with the environment. A common challenge is handling multi-modal action distributions, where multiple behaviours are represented in the data. Existing methods often assume unimodal behaviour policies, leading to suboptimal performance when this assumption is violated. We propose weighted imitation Learning on One Mode (LOM), a novel approach that focuses on learning from a single, promising mode of the behaviour policy. By using a Gaussian mixture model to identify modes and selecting the best mode based on expected returns, LOM avoids the pitfalls of averaging over conflicting actions. Theoretically, we show that LOM improves performance while maintaining simplicity in policy learning. Empirically, LOM outperforms existing methods on standard D4RL benchmarks and demonstrates its effectiveness in complex, multi-modal scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。