提出新型时序差分流模型,实现更长时展望的高精度未来状态预测。
Temporal Difference Flows
- 基于概率路径上的新贝尔曼方程与流匹配技术,直接学习未来状态分布。
- 在5倍以上长时展望下仍保持高质量生成,显著降低训练梯度方差。
- 适用于长期决策规划,尤其适合与预训练策略结合使用。
未来状态的预测对智能体推理与规划至关重要。传统方法通过逐步回放世界模型进行推演,易产生误差累积。几何时域模型(GHMs)通过直接预测未来状态提供替代方案,避免了累积误差。尽管可借助类似时序差分(TD)学习的生成方法训练,现有方法在训练中因自举预测而表现不佳,且难以生成长时展望下的高质量结果。本文提出时序差分流(TD-Flow),利用新颖的概率路径贝尔曼方程与流匹配技术,使GHM在超过5倍于先前方法的时展望长度上仍能准确学习。理论上,我们建立了新的收敛性结果,主要归因于训练期间梯度方差降低。进一步证明该思想可拓展至扩散类方法。实证上,我们在多个领域验证了TD-Flow在生成指标和下游任务(如策略评估)中的有效性。将TD-Flow与近期行为基础模型结合用于策略规划,性能显著提升,凸显其在长时决策中的潜力。
原文摘要 · Abstract (English)
Predictive models of the future are fundamental for an agent's ability to reason and plan. A common strategy learns a world model and unrolls it step-by-step at inference, where small errors can rapidly compound. Geometric Horizon Models (GHMs) offer a compelling alternative by directly making predictions of future states, avoiding cumulative inference errors. While GHMs can be conveniently learned by a generative analog to temporal difference (TD) learning, existing methods are negatively affected by bootstrapping predictions at train time and struggle to generate high-quality predictions at long horizons. This paper introduces Temporal Difference Flows (TD-Flow), which leverages the structure of a novel Bellman equation on probability paths alongside flow-matching techniques to learn accurate GHMs at over 5x the horizon length of prior methods. Theoretically, we establish a new convergence result and primarily attribute TD-Flow's efficacy to reduced gradient variance during training. We further show that similar arguments can be extended to diffusion-based methods. Empirically, we validate TD-Flow across a diverse set of domains on both generative metrics and downstream tasks including policy evaluation. Moreover, integrating TD-Flow with recent behavior foundation models for planning over pre-trained policies demonstrates substantial performance gains, underscoring its promise for long-horizon decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。