从像素中自动学习可独立控制的状态变量，提升强化学习效率
From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
- 用对比学习发现每个动作只影响部分状态变量的稀疏结构
- 在三个基准上直接从像素恢复出真实可控因子，性能超越现有方法
- 适合需要高效探索与可解释状态表示的研究者
利用因子化马尔可夫决策过程的算法比无视因子结构的方法样本效率更高，但其前提要求已知因子化表示——这一假设在智能体仅接收高维观测时失效。深度强化学习虽能处理此类输入，却无法利用因子结构。本文提出动作可控制因子化（ACF），一种基于对比学习的方法，可从像素观测中自动发现可独立控制的潜在变量：每个动作仅影响部分状态分量，其余变量由环境动态演化，形成对比训练的有用信号。ACF在三个具有已知因子结构的基准测试（Taxi、FourRooms、MiniGrid-DoorKey）上直接从像素恢复出真实可控因子，且持续优于基线解耦算法。
原文摘要 · Abstract (English)
Algorithms that exploit factored Markov decision processes are far more sample-efficient than factor-agnostic methods, yet they assume a factored representation is known a priori -- a requirement that breaks down when the agent sees only high-dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action-Controllable Factorization (ACF), a contrastive learning approach that uncovers independently controllable latent variables -- state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground truth controllable factors directly from pixel observations on three benchmarks with known factored structure -- Taxi, FourRooms, and MiniGrid-DoorKey -- consistently outperforming baseline disentanglement algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。