用音视频联合增强提升无监督动作定位效果
CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization
- 结合CLIP的跨模态协同增强，避免只关注显著区域
- 引入音频信息辅助判断动作边界，提升定位精度
- 无需额外标注，适合缺乏标签数据的场景
时间动作定位（TAL）在信息检索中备受关注。现有监督或弱监督方法严重依赖人工标注的时间边界和动作类别，成本高且耗时。因此，无监督时间动作定位（UTAL）逐渐流行。然而当前方法面临两大挑战：一是分类预训练特征过度聚焦于高度判别区域；二是仅依赖视觉信息难以确定上下文边界。为此，本文提出一种基于CLIP的跨视角音视频增强无监督动作定位方法。具体而言，引入视觉语言预训练与分类预训练协同增强机制，避免对显著区域的过度关注；同时融合音频感知，提供更丰富的上下文边界信息。最后，采用自监督跨视角学习范式，实现多模态感知增强而无需额外标注。在两个公开数据集上的大量实验表明，本模型优于多个先进方法。
原文摘要 · Abstract (English)
Temporal Action Localization (TAL) has garnered significant attention in information retrieval. Existing supervised or weakly supervised methods heavily rely on labeled temporal boundaries and action categories, which are labor-intensive and time-consuming. Consequently, unsupervised temporal action localization (UTAL) has gained popularity. However, current methods face two main challenges: 1) Classification pre-trained features overly focus on highly discriminative regions; 2) Solely relying on visual modality information makes it difficult to determine contextual boundaries. To address these issues, we propose a CLIP-assisted cross-view audiovisual enhanced UTAL method. Specifically, we introduce visual language pre-training (VLP) and classification pre-training-based collaborative enhancement to avoid excessive focus on highly discriminative regions; we also incorporate audio perception to provide richer contextual boundary information. Finally, we introduce a self-supervised cross-view learning paradigm to achieve multi-view perceptual enhancement without additional annotations. Extensive experiments on two public datasets demonstrate our model's superiority over several state-of-the-art competitors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。