用扩散模型生成前景知识,精准对齐视频与文本中的动作语义。
Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection

- 通过扩散过程去噪生成动作相关前景知识作为中间锚点。
- 在两个基准上达到当前最优性能,显著提升未见类动作检测精度。
- 适合关注开放词汇动作检测与跨模态对齐的研究者。
开放词汇时间动作检测(OV-TAD)旨在定位并分类未见类别动作段落,其中动作语义与视频表征的有效对齐至关重要。现有方法难以缓解抽象标签与复杂视频内容之间的语义失衡,导致引入语义噪声并误导跨模态对齐。为此,我们提出DFAlign,首个利用基于扩散的去噪机制生成前景知识以指导动作-视频对齐的框架。遵循‘条件化、去噪、对齐’流程,首先引入语义统一条件模块(SUC),将共享与特定动作语义统一为扩散去噪的条件;随后,背景抑制去噪模块(BSD)通过去噪过程逐步消除视频中的背景冗余,生成前景知识。该知识作为视频与文本表征间的有效中间语义锚点,缓解语义差距并增强动作相关片段的可区分性。进一步提出前景提示对齐模块(FPA),将提取的前景知识注入文本表征作为提示令牌,引导模型注意力聚焦于动作相关区域,实现精确跨模态对齐。大量实验表明,本方法在两个OV-TAD基准上均达到最先进性能。代码已公开:https://anonymous.4open.science/r/Code-2114/。
原文摘要 · Abstract (English)
Open-Vocabulary Temporal Action Detection (OV-TAD) aims to localize and classify action segments of unseen categories in untrimmed videos, where effective alignment between action semantics and video representations is critical for accurate detection. However, existing methods struggle to mitigate the semantic imbalance between concise, abstract action labels and rich, complex video contents, inevitably introducing semantic noise and misleading cross-modal alignment. To address this challenge, we propose DFAlign, the first framework that leverages diffusion-based denoising to generate foreground knowledge for the guidance of action-video alignment. Following the 'conditioning, denoising and aligning' manner, we first introduce the Semantic-Unify Conditioning (SUC) module, which unifies action-shared and action-specific semantics as conditions for diffusion denoising. Then, the Background-Suppress Denoising (BSD) module generates foreground knowledge by progressively removing background redundancy from videos through denoising process. This foreground knowledge serves as effective intermediate semantic anchor between video and text representations, mitigating the semantic gap and enhancing the discriminability of action-relevant segments. Furthermore, we introduce the Foreground-Prompt Alignment (FPA) module to inject extracted foreground knowledge as prompt tokens into text representations, guiding model's attention towards action-relevant segments and enabling precise cross-modal alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two OV-TAD benchmarks. The code repository is provided as follows: https://anonymous.4open.science/r/Code-2114/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。