用可微分维特比算法显式注入流程知识,让模型更高效地规划视频中的操作步骤。
ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videos
- 通过可微分维特比层将流程知识图谱融入规划过程,实现端到端训练。
- 参数量少一个数量级,且在三个数据集上达到顶尖性能。
- 适合需要高效、可解释规划的智能体应用,尤其关注样本效率与鲁棒性。
流程规划旨在预测从初始视觉状态到目标状态所需的一系列动作,是智能体在复杂环境中运行的核心能力。现有方法通常依赖大规模模型隐式学习流程结构,导致样本效率低且计算成本高。本文提出 ViterbiPlanNet,一种通过可微分维特比层(DVL)显式整合流程知识的框架。DVL 将流程知识图谱(PKG)直接嵌入维特比解码算法中,用平滑近似替代不可微操作,支持端到端优化,使模型通过基于图的解码进行学习。在 CrossTask、COIN 与 NIV 数据集上的实验表明,ViterbiPlanNet 在参数量少一个数量级的前提下,性能超越扩散模型与大语言模型驱动的规划器。大量消融实验证明性能提升源于结构感知的可微训练,而非事后修正,显著提升样本效率和对未见长序列的鲁棒性。我们还建立了统一测试协议,采用一致划分与评估指标,并通过多次实验与自助法分析确保结果统计显著性。
原文摘要 · Abstract (English)
Procedural planning aims to predict a sequence of actions that transforms an initial visual state into a desired goal, a fundamental ability for intelligent agents operating in complex environments. Existing approaches typically rely on large-scale models that learn procedural structures implicitly, resulting in limited sample-efficiency and high computational cost. In this work we introduce ViterbiPlanNet, a principled framework that explicitly integrates procedural knowledge into the learning process through a Differentiable Viterbi Layer (DVL). The DVL embeds a Procedural Knowledge Graph (PKG) directly with the Viterbi decoding algorithm, replacing non-differentiable operations with smooth relaxations that enable end-to-end optimization. This design allows the model to learn through graph-based decoding. Experiments on CrossTask, COIN, and NIV demonstrate that ViterbiPlanNet achieves state-of-the-art performance with an order of magnitude fewer parameters than diffusion- and LLM-based planners. Extensive ablations show that performance gains arise from our differentiable structure-aware training rather than post-hoc refinement, resulting in improved sample efficiency and robustness to shorter unseen horizons. We also address testing inconsistencies establishing a unified testing protocol with consistent splits and evaluation metrics. With this new protocol, we run experiments multiple times and report results using bootstrapping to assess statistical significance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。