用最优传输映射计算Wasserstein距离，稳定提升离线强化学习性能。
Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps
- 用输入凸神经网络建模最优传输映射，无需对抗训练
- 在D4RL上表现优于或相当主流方法，避免分布外动作
- 适合数据收集成本高的机器人等场景
离线强化学习旨在从静态数据集中学习最优策略，尤其适用于数据采集成本高的场景（如机器人）。其主要挑战是分布偏移：学习的策略偏离数据分布，可能导致不可靠的分布外动作。现有方法多采用密度比度量（如f-散度）进行正则化，本文提出一种基于Wasserstein距离的新方法，该距离对分布外数据更鲁棒，且能捕捉动作间的相似性。通过输入凸神经网络（ICNNs）建模最优传输映射，实现无需判别器的Wasserstein距离计算，避免了对抗训练，确保学习过程稳定。在D4RL基准数据集上，本方法表现与现有主流方法相当或更优。代码已公开于https://github.com/motokiomura/Q-DOT。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) aims to learn an optimal policy from a static dataset, making it particularly valuable in scenarios where data collection is costly, such as robotics. A major challenge in offline RL is distributional shift, where the learned policy deviates from the dataset distribution, potentially leading to unreliable out-of-distribution actions. To mitigate this issue, regularization techniques have been employed. While many existing methods utilize density ratio-based measures, such as the $f$-divergence, for regularization, we propose an approach that utilizes the Wasserstein distance, which is robust to out-of-distribution data and captures the similarity between actions. Our method employs input-convex neural networks (ICNNs) to model optimal transport maps, enabling the computation of the Wasserstein distance in a discriminator-free manner, thereby avoiding adversarial training and ensuring stable learning. Our approach demonstrates comparable or superior performance to widely used existing methods on the D4RL benchmark dataset. The code is available at https://github.com/motokiomura/Q-DOT .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。