用切片方法解决多维强化学习分布建模难题
Multivariate Distributional Reinforcement Learning Using Sliced Divergences
- 通过投影将一维散度扩展到多维回报分布
- 证明了均匀切片下贝尔曼算子的收缩性
- 适用于图像环境和雅达利游戏，支持多种散度
分布式强化学习（DRL）建模完整回报分布而非期望值，但扩展至多维情形仍具挑战。多数常用度量难以自然推广至高维或失去计算可处理性，且多维情形引入额外困难如一般矩阵折扣，现有无收缩性结果。本文提出切片分布强化学习（SDRL），通过投影将可计算的一维散度推广至多维回报分布。在共享标量折扣下，证明均匀切片的贝尔曼收缩性；提出最大切片变体，在一般稠密折扣矩阵下实现收缩性。SDRL支持广泛的基底散度，分析了Wasserstein、Cramér与最大均值差异（MMD），并明确哪些变体适配标准单样本贝尔曼更新。在简单链问题、基于图像的网格世界环境及部分Atari游戏中评估了SDRL性能。
原文摘要 · Abstract (English)
Distributional reinforcement learning (DRL) models the full return distribution rather than expectations, but extending it to multivariate settings remains challenging. Many common metrics do not naturally generalize beyond one dimension or lose computational tractability, and the multivariate case introduces additional difficulties such as general matrix discounting, for which no contraction results are available. We introduce Sliced Distributional Reinforcement Learning (SDRL), which lifts tractable one-dimensional divergences to multivariate return distributions via projections. We prove Bellman contraction for uniform slicing under shared scalar discounting, and introduce a maximum-slicing variant with contraction under general dense discount matrices. SDRL supports a broad class of base divergences; we analyze Wasserstein, Cramér, and Maximum Mean Discrepancy (MMD), and characterize which SDRL variants suit the standard single-sample Bellman update used in distributional RL. We evaluate SDRL on a toy chain problem and a gridworld image-based environment as well as a subset of Atari games.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。