arXiv:2606.25170stat.MLcs.LG2026-06

提出上下文马尔可夫决策模型的样本高效学习方法,突破上下文维度瓶颈。

Minimax PAC Bounds for Learning in Exogenous Contextual MDPs

  • 基于上下文采样构造方差缩减算法,实现高精度策略评估
  • 在未知环境下保持上下文无关的样本复杂度,上界与下界匹配
  • 适用于上下文独立、状态空间有限的强化学习场景

研究具有外生独立同分布上下文的表格化折扣马尔可夫决策过程中的PAC学习问题,其中上下文由未知分布μ独立生成并在动作前揭示。上下文影响奖励与转移,但不受智能体控制。根据信息获取方式,学习者可访问μ的采样器、给定状态-上下文-动作三元组的转移核采样器,或两者兼有。样本复杂度以执行前(n)和执行中(m)调用采样器次数衡量。当奖励与转移已知仅需采样μ时,提出方差缩减算法,在(~O(1/((1−γ)³ε²)), 0)样本复杂度下完成策略评估、最优值估计与最优策略提取,且复杂度与|Z|无关,达到最小最大最优(对数因子内)。作为推论,也获得一步完美前瞻情形下的紧致率。在完全未知情形下,仍保持|Z|-无关的策略评估复杂度,上下界均为(~O(|X|/((1−γ)³ε²)), ~O(1/((1−γ)²ε²)))。

原文摘要 · Abstract (English)

We study PAC learning in tabular discounted Markov decision processes with exogenous i.i.d. contexts, with discount factor $γ$, finite state space $\mathcal X$, action space $\mathcal A$, and context space $\mathcal Z$. At each time step, a context is drawn independently from an unknown distribution $μ$ and revealed before the agent acts. This context may affect both rewards and transitions, while remaining uncontrolled by the agent. Depending on the regime, the learner has access either to a sampling oracle for $μ$, to a sampling oracle for the transition kernel conditioned on state-context-action tuples, or to both. Oracles can be accessed before and during policy execution. The sample complexity is measured by a couple $(n,m)$, where $n$ is the number of calls to the sampling oracles before execution and $m$ is the number of calls to the sampling oracles during execution. When rewards and transitions are known and only the context distribution $μ$ is sampled, we give a variance-reduced algorithm that solves policy evaluation (PE), best-value estimation (BVE), and best-policy extraction (BPE) with $\left(\widetilde O\left(1/((1-γ)^3\varepsilon^2)\right), 0 \right) $ sample complexity. The rate is independent of $|\mathcal Z|$ and minimax optimal up to logarithmic factors. As a corollary, we also obtain tight rates in the case of one-step perfect look-ahead, improving upon the existing guarantees. In the fully unknown regime, where both $μ$ and P must be learned, we show that PE remains $|\mathcal Z|$-free, with matching upper and lower bounds $\bigl(\widetilde O(|\mathcal X|/((1-γ)^3\varepsilon^2)),\, \widetilde O(1/((1-γ)^2\varepsilon^2))\bigr)$.

强化学习采样复杂度上下文最优性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。