找出让后续任务表现最优的通用状态特征,突破传统方法局限。
Which Features are Best for Successor Features?
- 基于下游任务性能优化,推导出三类任务共用的最优基特征
- 最优特征为Δ⁻¹ + (Δ⁻¹)⁎的特征函数,非传统拉普拉斯特征函数
- 适用于离线强化学习中的行为克隆正则化场景,理论意义强
在强化学习中,通用后续特征(SFs)可实现测试时零样本适应新任务:对线性组合于一组基础特征的任意下游奖励函数,均能提供最优策略。但何为优质的基础特征仍不明确,尤其在超出其线性跨度的任务上。已有研究指出拉普拉斯特征函数(Δ+Δ⁎的特征函数,Δ为参考策略的拉普拉斯算子,Δ⁎为时间反演动力学的拉普拉斯算子)具有潜力并表现出良好实证性能。本文首次在非循环定义下,基于下游性能的客观标准,识别出最优基础特征,无需假设下游任务为特征的线性组合。我们针对三类典型任务:到达随机目标状态、密集随机高斯奖励、随机稀疏奖励,进行分析。结果表明,三类任务在期望下游性能上达到最优的特征完全一致。这些特征一般并非拉普拉斯特征函数,但在最简情形(确定性环境且折扣因子γ接近1)下,它们是Δ⁻¹+(Δ⁻¹)⁎的特征函数。该结论建立在大行为克隆正则化假设之上,该设定常用于离线强化学习。研究过程中,还揭示了KL正则化自然策略梯度的新性质,以及贝尔曼残差范数无法反映后续特征信息的本质原因。
原文摘要 · Abstract (English)
In reinforcement learning, universal successor features (SFs) are a way to provide zero-shot adaptation to new tasks at test time: they provide optimal policies for all downstream reward functions lying in the linear span of a set of base features. But it is unclear what constitutes a good set of base features, that could be useful for a wide set of downstream tasks beyond their linear span. Laplacian eigenfunctions (the eigenfunctions of $Δ+Δ^\ast$ with $Δ$ the Laplacian operator of some reference policy and $Δ^\ast$ that of the time-reversed dynamics) have been argued to play a role, and offer good empirical performance. Here, for the first time, we identify the optimal base features based on an objective criterion of downstream performance, in a non-tautological way without assuming the downstream tasks are linear in the features. We do this for three generic classes of downstream tasks: reaching a random goal state, dense random Gaussian rewards, and random ``scattered'' sparse rewards. The features yielding optimal expected downstream performance turn out to be the \emph{same} for these three task families. They do not coincide with Laplacian eigenfunctions in general, though they can be expressed from $Δ$: in the simplest case (deterministic environment and decay factor $γ$ close to $1$), they are the eigenfunctions of $Δ^{-1}+(Δ^{-1})^\ast$. We obtain these results under an assumption of large behavior cloning regularization with respect to a reference policy, a setting often used for offline RL. Along the way, we get new insights into KL-regularized\option{natural} policy gradient, and into the lack of SF information in the norm of Bellman gaps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。