用扩散模型生成视频步骤,让动作衔接更自然。
Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos
- 在扩散模型中引入可学习插值矩阵,生成中间状态特征。
- 在三个数据集上优于现有方法,尤其提升起止状态附近预测精度。
- 适合需要精准步骤规划的视频理解任务,如教学视频分析。
本文针对教学视频中的流程规划问题,旨在从起始和终止视觉观测中生成连贯且任务对齐的动作序列。以往工作主要依赖文本监督来弥合观察状态与未观测动作之间的差距,但难以捕捉动作间的复杂时序关系。为此,我们提出掩码时序插值扩散(MTID)模型,在扩散模型中引入潜在空间时序插值模块,利用可学习插值矩阵生成中间潜在特征,从而以更丰富的中间状态细节增强视觉监督。通过将此增强监督融入模型,实现面向特定任务需求的端到端训练,显著提升模型对时序一致动作序列的预测能力。此外,我们设计了动作感知掩码投影机制,限制动作生成空间,并引入任务自适应掩码邻近损失,优先保证起止状态附近的预测准确性,同时过滤无关动作,生成上下文感知的动作序列。在三个主流基准数据集上的实验结果表明,MTID在多数指标上表现优异。代码已公开于 https://github.com/WiserZhou/MTID。
原文摘要 · Abstract (English)
In this paper, we address the challenge of procedure planning in instructional videos, aiming to generate coherent and task-aligned action sequences from start and end visual observations. Previous work has mainly relied on text-level supervision to bridge the gap between observed states and unobserved actions, but it struggles with capturing intricate temporal relationships among actions. Building on these efforts, we propose the Masked Temporal Interpolation Diffusion (MTID) model that introduces a latent space temporal interpolation module within the diffusion model. This module leverages a learnable interpolation matrix to generate intermediate latent features, thereby augmenting visual supervision with richer mid-state details. By integrating this enriched supervision into the model, we enable end-to-end training tailored to task-specific requirements, significantly enhancing the model's capacity to predict temporally coherent action sequences. Additionally, we introduce an action-aware mask projection mechanism to restrict the action generation space, combined with a task-adaptive masked proximity loss to prioritize more accurate reasoning results close to the given start and end states over those in intermediate steps. Simultaneously, it filters out task-irrelevant action predictions, leading to contextually aware action sequences. Experimental results across three widely used benchmark datasets demonstrate that our MTID achieves promising action planning performance on most metrics. The code is available at https://github.com/WiserZhou/MTID.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。