用扩散模型改进动作定位,让时间边界和语义一起优化。
Masked Diffusion Vision-Language Models for Temporal Action Localization

- 用双向注意力的扩散模型同时优化时间边界和语义内容
- 在ActivityNet-RTL上mAP提升12.3%,严格阈值下效果更优
- 适合需要精准时间定位的视频理解任务
时序动作定位(TAL)需在未剪辑视频中识别目标事件并精确定位其起止时间。现有视觉语言模型虽增强语义推理与语言引导输出,但自回归解码器仍从左到右生成,无法利用后续语义修正早期时间预测。本文将掩码扩散视觉语言模型(MDVLMs)适配至TAL,使语义标记与边界标记在迭代去噪过程中始终保持可编辑,支持双向注意力下的联合优化。直接适配存在两个问题:标准掩码训练随机丢弃所有位置,而时间标记在语义充足时更可靠;标记级交叉熵不反映时间重叠度。为此,提出计划性训练目标,采用边界感知掩码与步长加权重建,模拟后期时间标记恢复过程,并引入步级交并比奖励,在去噪阶段提供重叠感知监督。标准序列级交叉熵提供基础重建信号。在ActivityNet-RTL、ActivityNet-1.3和THUMOS-14上的实验表明,MDVLM-TAL在时序推理与边界定位上均优于自回归基线模型,尤其在严格交并比标准下表现突出。
原文摘要 · Abstract (English)
Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos. Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right, preventing later semantic evidence from revising earlier timestamp predictions. We adapt masked diffusion vision-language models (MDVLMs) to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising with bidirectional attention, allowing temporal boundaries and semantic content to be refined jointly. Direct adaptation, however, creates two TAL-specific mismatches: standard masked diffusion training corrupts all positions uniformly at random, but the time tokens are more reliable when enough semantic context is available; and token-level cross-entropy does not reflect temporal IoU. To address these mismatches, we introduce a Planned Training Objective that uses boundary-aware masking and step-weighted reconstruction to rehearse the late recovery of time tokens, together with a Step-Level IoU Reward that provides overlap-aware supervision during denoising. A standard sequence-level cross-entropy term provides the base reconstruction signal. Experiments on ActivityNet-RTL, ActivityNet-1.3, and THUMOS-14 show that MDVLM-TAL improves both temporal reasoning and boundary localization over autoregressive vision-language baselines, with especially strong gains under stricter temporal IoU criteria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。