将状态转移视为三阶张量,分解出独立的状态、动作、下一状态编码器。
Factorized Spectral Representations for Reinforcement Learning

- 用CP分解对转移张量建模,分别学习状态、动作、下一状态的特征映射。
- 在高维运动任务上样本效率提升,需样本量随状态或动作维度缩小。
- 状态编码可跨执行器迁移,仅需重训练动作编码,适合动态结构相似场景。
从交互数据中学习紧凑的世界模型是实现高效深度强化学习的核心。谱表示方法通过将转移核视为状态-动作对与下一状态之间的矩阵,并利用自监督对比目标进行低秩分解,已成为连续控制中的主流范式。本文进一步深化这一视角:转移核天然为状态、动作和下一状态构成的三阶张量,其CP分解可产生每种模式对应的特征映射。我们提出FaStR,通过噪声对比目标拟合该分解,生成独立的状态、动作和下一状态编码器,共同构成统一的谱表示。因子化形式降低了假设空间复杂度,所需样本量随较小的维度(状态或动作)缩放而减少。实验表明,FaStR在高维运动任务上表现最优,且学习到的状态编码可完整迁移至不同执行器,仅需重训练动作编码器。
原文摘要 · Abstract (English)
Learning a compact model of the world from interaction data is central to sample-efficient deep reinforcement learning. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state-action pairs on one side and next states on the other, and learning a low-rank factorization through self-supervised contrastive objectives. We take this view one step further. The transition kernel is naturally a three-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next-state encoders that together form a single spectral representation. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions. Empirically, FaStR delivers its largest gains on high-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。