arXiv:2603.17577cs.LGcs.AI2026-03

仅凭无动作轨迹与演示者身份,就能识别隐藏动作与环境动态。

Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversity

  • 利用不同演示者的策略差异,将观测转移分布分解为潜在动作条件的混合模型。
  • 在策略多样性足够且满足秩条件时,可唯一还原潜行动作与环境动态。
  • 适用于缺乏动作标签的离线强化学习,尤其适合策略差异明显的场景。

当轨迹中未观测到动作但标注了演示者身份时,能否恢复潜行动作与环境动态?我们在此设定下研究该问题:每名演示者遵循不同策略,环境动态共享,且演示者身份仅通过所选动作影响下一观测。此时,条件观测分布 $p(o_{t+1}ackslashmid o_t,e)$ 是潜行动作条件转移核的混合,混合权重由演示者决定。这导致每个状态下的可观测转移分布可进行列随机非负矩阵分解。在足够分散的策略多样性和秩条件下,我们证明潜移动态与演示者策略可被唯一识别,仅存在潜行动作标签的排列歧义。我们进一步通过格拉姆行列式最小体积准则将结果扩展至连续观测空间,并表明在连通状态空间上,转移映射的连续性可将局部排列歧义升级为单一全局排列。少量带标签动作数据即可消除最终歧义。这些结果确立了演示者多样性作为从离线强化学习数据中识别潜行动作与动态的可解释性来源。

原文摘要 · Abstract (English)

Can latent actions and environment dynamics be recovered from offline trajectories when actions are never observed? We study this question in a setting where trajectories are action-free but tagged with demonstrator identity. We assume that each demonstrator follows a distinct policy, while the environment dynamics are shared across demonstrators and identity affects the next observation only through the chosen action. Under these assumptions, the conditional next-observation distribution $p(o_{t+1}\mid o_t,e)$ is a mixture of latent action-conditioned transition kernels with demonstrator-specific mixing weights. We show that this induces, for each state, a column-stochastic nonnegative matrix factorization of the observable conditional distribution. Using sufficiently scattered policy diversity and rank conditions, we prove that the latent transitions and demonstrator policies are identifiable up to permutation of the latent action labels. We extend the result to continuous observation spaces via a Gram-determinant minimum-volume criterion, and show that continuity of the transition map over a connected state space upgrades local permutation ambiguities to a single global permutation. A small amount of labeled action data then suffices to fix this final ambiguity. These results establish demonstrator diversity as a principled source of identifiability for learning latent actions and dynamics from offline RL data.

离线RL潜行动作可识别性策略差异

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。