仅凭最优动作，可推断马尔可夫决策过程的转移规律边界。
From Optimal Actions to World Models: Identifiability of Transition Kernels in Discounted MDPs
- 通过最优动作反推转移核的可识别性条件
- 状态-动作奖励下转移核有n(n−1)维不可分辨族
- 仅依赖状态奖励时几乎无法还原动力学
我们研究仅从最优动作中能恢复多少马尔可夫决策过程的转移概率。该问题与Letcher等人提出的逆问题密切相关，后者关注是否能从数值$Q$值中恢复动态系统。但本文不观察$Q$值本身，仅知每类奖励下的最优动作。对于状态-动作奖励 $r(s,a)$，知道所有奖励下的最优动作可揭示某一动作相对于另一动作在跟随同一固定策略时的优劣程度。然而这仍不足以唯一确定转移概率。我们证明：两个转移核对所有奖励产生相同最优动作，当且仅当满足 \\[ Q_{s,a} = \Bigl(P_{s,a}+\tfrac1γe_s^\mathsf T(L-I)\Bigr)L^{-1} \\[ 其中 $L$ 为满足 $L\mathbf 1=\mathbf 1$ 的可逆矩阵。在具有严格正元素的转移核附近，存在一个 $n(n-1)$ 维的不同转移核族具有此性质。即使只考虑每个状态有唯一最优动作的情形，结论不变。进一步比较 $r(s)$ 与 $r(s,a,s')$ 形式的奖励：依赖下一状态的奖励通常可完全恢复转移核，除单动作状态行可能隐藏外；而仅依赖状态的奖励则信息更少：两个核产生相同最优动作集当且仅当每个确定性策略对同一组奖励均最优。结果表明奖励形式显著影响仅从最优动作中可学习的动力学结构。
原文摘要 · Abstract (English)
We study what can be recovered about the transition probabilities of a Markov decision process from optimal actions alone. This is closely related to the inverse problem considered by Letcher et al., who ask when the dynamics can be recovered from numerical \(Q\)-values. Here the numerical values themselves are not observed; only the optimal actions are known, for every reward in a given class. For state-action rewards \(r(s,a)\), knowing the optimal actions for every reward also tells us how much better one action is than another when each is followed by the same fixed policy. This is still not enough to determine the transition probabilities uniquely. We prove that two kernels give the same optimal actions for every reward exactly when \[ Q_{s,a} = \Bigl(P_{s,a}+\tfrac1γe_s^{\mathsf T}(L-I)\Bigr)L^{-1} \] for one invertible matrix \(L\) satisfying \(L\mathbf 1=\mathbf 1\). Near a kernel with strictly positive entries, there is an \(n(n-1)\)-dimensional family of different kernels with this property. The result is unchanged if we consider only rewards having a unique optimal action at every state. We then compare this with rewards of the forms \(r(s)\) and \(r(s,a,s')\). Rewards that depend on the next state can usually recover the transition kernel itself: every row at a state with at least two actions is determined, and we describe exactly when a row at a state with one action can remain hidden. State rewards reveal less: two kernels give the same optimal actions exactly when every deterministic policy is optimal for the same set of rewards. The results show how the form of the reward affects what can be learned about the dynamics from optimal actions alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。