arXiv:2510.05446cs.LGstat.ML2025-10被引 4

用学习到的先验提升元强化学习,让算法更快适应新任务。

Prior-Aligned Meta-RL: Thompson Sampling with Learned Priors and Guarantees in Finite-Horizon MDPs

  • 基于线性表示和高斯先验,设计两类汤普森采样算法。
  • 在有限任务下实现优于传统方法的元后悔上界,任务越多越优。
  • 适合有大量相似任务、需快速适应的推荐与决策场景。

我们研究有限时域马尔可夫决策过程中的元强化学习,其中相关任务在其最优动作价值函数结构上具有相似性。具体地,假设 $Q^*_h(s,a)=Φ_h(s,a) heta^{(k)}_h$ 的线性表示,并在任务特定参数 $θ^{(k)}_h$ 上设定高斯元先验 $\mathcal{N}(θ^*_h,Σ^*_h)$。基于随机值函数,我们提出两种汤普森风格算法:(i) MTSRL,仅学习先验均值,使用学习到的均值与已知协方差进行后验采样;(ii) $\text{MTSRL}^{+}$,额外估计协方差并采用先验扩宽控制有限样本误差。进一步,我们提出先验对齐技术,将学习先验下的后验与已知真实先验的元预言机耦合,获得元后悔保证:在小任务规模下匹配无先验的汤普森采样,在更多任务时严格改进。具体而言,已知协方差下获得 $\tilde{O}(H^{4}S^{3/2}\sqrt{ANK})$ 元后悔,学习协方差下为 $\tilde{O}(H^{4}S^{3/2}\sqrt{AN^3K})$;两者分别在 $K \gtrsim \tilde{O}(H^2)$ 与 $K \gtrsim \tilde{O}(N^2H^2)$ 后优于无先验方法。在含特征与先验误设的状态感知推荐环境中模拟显示,经过短暂探索后,MTSRL/MTSRL extsuperscript{+} 跟踪元预言机,显著超越无先验强化学习与仅带基线的元贝叶斯方法。结果首次提供了具有学习 Q-先验的汤普森风格强化学习的元后悔保证,并为实验丰富场景提供实用方案(如通过 RLSVI 暖启动、OLS 聚合、协方差扩宽)。

原文摘要 · Abstract (English)

We study meta-reinforcement learning in finite-horizon MDPs where related tasks share similar structures in their optimal action-value functions. Specifically, we posit a linear representation $Q^*_h(s,a)=Φ_h(s,a)\,θ^{(k)}_h$ and place a Gaussian meta-prior $ \mathcal{N}(θ^*_h,Σ^*_h)$ over the task-specific parameters $θ^{(k)}_h$. Building on randomized value functions, we propose two Thompson-style algorithms: (i) MTSRL, which learns only the prior mean and performs posterior sampling with the learned mean and known covariance; and (ii) $\text{MTSRL}^{+}$, which additionally estimates the covariance and employs prior widening to control finite-sample estimation error. Further, we develop a prior-alignment technique that couples the posterior under the learned prior with a meta-oracle that knows the true prior, yielding meta-regret guarantees: we match prior-independent Thompson sampling in the small-task regime and strictly improve with more tasks once the prior is learned. Concretely, for known covariance we obtain $\tilde{O}(H^{4}S^{3/2}\sqrt{ANK})$ meta-regret, and with learned covariance $\tilde{O}(H^{4}S^{3/2}\sqrt{AN^3K})$; both recover a better behavior than prior-independent after $K \gtrsim \tilde{O}(H^2)$ and $K \gtrsim \tilde{O}(N^2H^2)$, respectively. Simulations on a stateful recommendation environment (with feature and prior misspecification) show that after brief exploration, MTSRL/MTSRL\(^+\) track the meta-oracle and substantially outperform prior-independent RL and bandit-only meta-baselines. Our results give the first meta-regret guarantees for Thompson-style RL with learned Q-priors, and provide practical recipes (warm-start via RLSVI, OLS aggregation, covariance widening) for experiment-rich settings.

元强化学习汤普森采样先验学习后悔上界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。