提出一种新型强化学习表征,可高效实现奖励加权轨迹建模。
The Terminal Representation in Reinforcement Learning

- 基于奖励加权轨迹构建低维表征,无需特征分解
- 理论证明两种学习算法收敛,支持零样本组合性
- 比传统方法更省计算资源,适合实际部署
表示学习是强化学习中时空抽象的强大工具。现有主流方法包括后续表示(SR)和默认表示(DR):SR通过未来轨迹编码状态,解耦信息流与奖励;DR在此基础上用奖励加权轨迹,将信用分配结构融入表示。二者的特征向量已用于选项发现、奖励塑造、迁移学习和探索等任务。本文提出结构不同的终端表示(TR),其同样基于奖励加权轨迹,但可作为低维对象学习,并直接用于上述应用,无需特征分解。该方法摆脱了特征分解对对称转移动态的假设。本文建立TR的理论基础:推导过程、两类学习算法的收敛性、零样本组合性应用,以及不同奖励形式间的等价性。进一步表明,TR嵌入在顶层DR特征向量中,可无损捕获相同知识。实验证明,TR在子任务中可替代现有表示,且学习、存储和使用成本更低。
原文摘要 · Abstract (English)
Representation learning is a powerful tool for spatio-temporal abstraction within reinforcement learning (RL). Two well established approaches are through the successor representation (SR) and the default representation (DR). The SR encodes states by the future trajectories they induce, capturing information flow decoupled from reward. The DR builds on this by weighting trajectories with reward, integrating credit-assignment structure into the representation. Eigenvectors of both representations have been used to support a range of downstream tasks -- including option discovery, reward shaping, transfer learning, and exploration. We introduce a structurally distinct formulation: the terminal representation (TR). The TR encodes reward-weighted trajectories similarly to the DR, but can be learned as a lower-dimensionality object, and can be used directly for the mentioned applications without eigenvector computations. Eigendecomposition also imposes the assumption of symmetric transition dynamics, which the TR can bypass. In this work we develop the theoretical foundations of the TR: its derivation, convergence of two learning algorithms, its use for zero-shot compositionality, and equivalences between alternative reward formulations. We further show the TR is embedded in the top DR eigenvector, allowing it to capture the same underlying knowledge without eigendecomposition. Additionally, we provide empirical evidence of the TR as a viable alternative to existing representations in subsidiary applications, while requiring less computational overhead to learn, store, and use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。