提出新算法，让离线强化学习生成安全高效的稀疏策略。
Fat-to-Thin Policy Optimization: Offline RL with Sparse Policies
- 用胖（重尾）提议策略指导瘦（稀疏）策略学习。
- 在医疗模拟和MuJoCo任务中表现优于现有方法。
- 适合需要高安全性、动作选择稀疏的现实场景。
稀疏连续策略是指某些动作可随机选择，而其他动作概率严格为零的分布，与高斯分布截然不同，对安全关键任务（如医学治疗）具有重要意义。将离线强化学习与稀疏策略结合，可仅从历史数据中学习到具备安全意识的稀疏策略。然而，现有离线算法难以处理稀疏策略中不在当前支持集内的动作评估问题。本文提出首个解决该挑战的离线策略优化算法：胖-瘦策略优化（FtTPO）。具体而言，维护一个胖（重尾）提议策略以从数据中有效学习，并向负责环境交互的瘦（稀疏）策略注入知识。我们采用广义q-高斯族实例化FtTPO，该族涵盖重尾与稀疏策略。实验验证其在安全关键治疗仿真及标准MuJoCo基准上均表现优异。代码已公开于https://github.com/lingweizhu/fat2thin。
原文摘要 · Abstract (English)
Sparse continuous policies are distributions that can choose some actions at random yet keep strictly zero probability for the other actions, which are radically different from the Gaussian. They have important real-world implications, e.g. in modeling safety-critical tasks like medicine. The combination of offline reinforcement learning and sparse policies provides a novel paradigm that enables learning completely from logged datasets a safety-aware sparse policy. However, sparse policies can cause difficulty with the existing offline algorithms which require evaluating actions that fall outside of the current support. In this paper, we propose the first offline policy optimization algorithm that tackles this challenge: Fat-to-Thin Policy Optimization (FtTPO). Specifically, we maintain a fat (heavy-tailed) proposal policy that effectively learns from the dataset and injects knowledge to a thin (sparse) policy, which is responsible for interacting with the environment. We instantiate FtTPO with the general $q$-Gaussian family that encompasses both heavy-tailed and sparse policies and verify that it performs favorably in a safety-critical treatment simulation and the standard MuJoCo suite. Our code is available at \url{https://github.com/lingweizhu/fat2thin}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。