利用多智能体数据差异,从无动作视频中学习可控特征表示。
Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasets
- 通过比较不同策略智能体的无动作轨迹,提取可控特征
- 在存在时序相关噪声的环境中实现可学习的低维状态表示
- 适合有多个异策略视频数据的离线强化学习场景
序列决策环境常包含高维观测,其中部分特征与智能体控制无关,构成不可控噪声,增加状态空间复杂度。为在可处理的低维状态空间中学习有效策略,需忽略这些噪声特征。由于此类环境视频数据丰富,基于无动作离线数据的任务无关表征学习具有吸引力。然而,现有研究指出,在外生块马尔可夫决策过程(Ex-BMDP)模型下,当观测中存在时序相关噪声时,无动作学习面临理论局限。本文提出,在多个具有不同策略的智能体提供的无动作视频数据可用时,该问题变得可解。为此,我们提出CRAFT(基于轨迹比较的无动作表征),一种样本高效算法,利用不同智能体间可控特征动态的差异进行表征学习。我们提供了CRAFT性能的理论保证,并在简单示例中验证其可行性,为类似设置下的实用方法奠定基础。
原文摘要 · Abstract (English)
While sequential decision-making environments often involve high-dimensional observations, not all features of these observations are relevant for control. In particular, the observation space may capture factors of the environment which are not controllable by the agent, but which add complexity to the observation space. The need to ignore these "noise" features in order to operate in a tractably-small state space poses a challenge for efficient policy learning. Due to the abundance of video data available in many such environments, task-independent representation learning from action-free offline data offers an attractive solution. However, recent work has highlighted theoretical limitations in action-free learning under the Exogenous Block MDP (Ex-BMDP) model, where temporally-correlated noise features are present in the observations. To address these limitations, we identify a realistic setting where representation learning in Ex-BMDPs becomes tractable: when action-free video data from multiple agents with differing policies are available. Concretely, this paper introduces CRAFT (Comparison-based Representations from Action-Free Trajectories), a sample-efficient algorithm leveraging differences in controllable feature dynamics across agents to learn representations. We provide theoretical guarantees for CRAFT's performance and demonstrate its feasibility on a toy example, offering a foundation for practical methods in similar settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。