不同强化学习方法学出的表示受目标影响，揭示了算法本质差异
Task-Induced Representational Invariances Depend on Learning Objective in Deep RL
- 用马尔可夫决策过程简化理论分析深度强化学习表示
- DQN学得对状态对称性不变的表示，PPO则对动作对称性不变
- 结果可用于算法比较、迁移学习优化，或启发脑神经编码研究
强化学习（RL）长期被用作神经科学中目标导向动物行为的模型。现代深度强化学习在多个领域取得显著成功，进一步加强了这一联系。学习高维状态空间的抽象表示是其成功的关键。然而，对这些学习表示的理论理解仍不充分，阻碍了模型与动物学习之间的直接比较。本文通过马尔可夫决策过程（MDP）简化理论分析深度强化学习表示。在导航任务中研究经典强化学习算法，发现即使性能相当，基于价值的方法（DQN）学得对MDP同构对称性不变的表示，而基于策略梯度的方法（PPO）学得对动作对称性不变的表示。这些差异在多个领域中一致出现，对迁移学习有下游影响，并在大语言模型中呈现提示依赖性。我们的发现为跨算法表示比较提供了原则性方法，具有实际应用意义，可能为大脑神经编码提供新见解。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has long served as a model for goal-directed animal behavior in neuroscience. Modern deep RL has shown remarkable success across many domains, further strengthening this connection. The ability to learn abstract representations of high-dimensional state spaces underlies much of this success. However, theoretical understanding of these learned representations remains limited, hindering direct comparisons between models and animal learning. We address this gap by analyzing deep RL representations through the lens of MDP reduction theory. Investigating canonical RL algorithms in a navigation task, we find that even when performance is comparable, the value-based method (DQN) learns representations that are invariant to MDP homomorphism symmetries, while the policy-gradient method (PPO) learns representations invariant to action symmetries. These differences emerge consistently across domains, have downstream consequences for transfer learning, and appear in LLMs in a prompt-dependent manner. Our findings provide a principled approach to comparing learned representations across RL algorithms, with demonstrated practical implications and possible insights for neural coding in the brain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。