arXiv:2502.10790cs.LGmath.OC2025-02被引 2

找出让后续任务表现最优的通用状态特征,突破传统方法局限。

Which Features are Best for Successor Features?

  • 基于下游任务性能优化,推导出三类任务共用的最优基特征
  • 最优特征为Δ⁻¹ + (Δ⁻¹)⁎的特征函数,非传统拉普拉斯特征函数
  • 适用于离线强化学习中的行为克隆正则化场景,理论意义强

在强化学习中,通用后续特征(SFs)可实现测试时零样本适应新任务:对线性组合于一组基础特征的任意下游奖励函数,均能提供最优策略。但何为优质的基础特征仍不明确,尤其在超出其线性跨度的任务上。已有研究指出拉普拉斯特征函数(Δ+Δ⁎的特征函数,Δ为参考策略的拉普拉斯算子,Δ⁎为时间反演动力学的拉普拉斯算子)具有潜力并表现出良好实证性能。本文首次在非循环定义下,基于下游性能的客观标准,识别出最优基础特征,无需假设下游任务为特征的线性组合。我们针对三类典型任务:到达随机目标状态、密集随机高斯奖励、随机稀疏奖励,进行分析。结果表明,三类任务在期望下游性能上达到最优的特征完全一致。这些特征一般并非拉普拉斯特征函数,但在最简情形(确定性环境且折扣因子γ接近1)下,它们是Δ⁻¹+(Δ⁻¹)⁎的特征函数。该结论建立在大行为克隆正则化假设之上,该设定常用于离线强化学习。研究过程中,还揭示了KL正则化自然策略梯度的新性质,以及贝尔曼残差范数无法反映后续特征信息的本质原因。

原文摘要 · Abstract (English)

In reinforcement learning, universal successor features (SFs) are a way to provide zero-shot adaptation to new tasks at test time: they provide optimal policies for all downstream reward functions lying in the linear span of a set of base features. But it is unclear what constitutes a good set of base features, that could be useful for a wide set of downstream tasks beyond their linear span. Laplacian eigenfunctions (the eigenfunctions of $Δ+Δ^\ast$ with $Δ$ the Laplacian operator of some reference policy and $Δ^\ast$ that of the time-reversed dynamics) have been argued to play a role, and offer good empirical performance. Here, for the first time, we identify the optimal base features based on an objective criterion of downstream performance, in a non-tautological way without assuming the downstream tasks are linear in the features. We do this for three generic classes of downstream tasks: reaching a random goal state, dense random Gaussian rewards, and random ``scattered'' sparse rewards. The features yielding optimal expected downstream performance turn out to be the \emph{same} for these three task families. They do not coincide with Laplacian eigenfunctions in general, though they can be expressed from $Δ$: in the simplest case (deterministic environment and decay factor $γ$ close to $1$), they are the eigenfunctions of $Δ^{-1}+(Δ^{-1})^\ast$. We obtain these results under an assumption of large behavior cloning regularization with respect to a reference policy, a setting often used for offline RL. Along the way, we get new insights into KL-regularized\option{natural} policy gradient, and into the lack of SF information in the norm of Bellman gaps.

强化学习状态特征最优基函数离线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。