精准定位长视频中重叠动作，提升多模态理解精度
DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding
- 用自注意力对齐音视频特征，增强模态内一致性
- 跨模态多尺度交互建模，提升细粒度动作识别准确率
- 在多个真实数据集上刷新性能，适合复杂视频分析场景
现实世界视频常包含重叠事件和复杂的时序依赖，使得多模态交互建模尤为困难。我们提出DEL框架，用于密集语义动作定位，旨在在长未剪辑视频中以细粒度时间分辨率准确检测并分类多个动作。DEL包含两个核心模块：利用掩码自注意力对齐音视频特征以增强模态内一致性；以及多尺度跨模态交互优化模块，建模跨模态依赖关系，实现高层语义与细粒度细节的融合。该方法在多个真实世界时序动作定位数据集（UnAV-100、THUMOS14、ActivityNet 1.3、EPIC-Kitchens-100）上达到领先性能，相比此前方法分别取得+3.3%、+2.6%、+1.2%、+1.7%（动词）、+1.4%（名词）的平均mAP提升。
原文摘要 · Abstract (English)
Real-world videos often contain overlapping events and complex temporal dependencies, making multimodal interaction modeling particularly challenging. We introduce DEL, a framework for dense semantic action localization, aiming to accurately detect and classify multiple actions at fine-grained temporal resolutions in long untrimmed videos. DEL consists of two key modules: the alignment of audio and visual features that leverage masked self-attention to enhance intra-mode consistency and a multimodal interaction refinement module that models cross-modal dependencies across multiple scales, enabling high-level semantics and fine-grained details. Our method achieves state-of-the-art performance on multiple real-world Temporal Action Localization (TAL) datasets, UnAV-100, THUMOS14, ActivityNet 1.3, and EPIC-Kitchens-100, surpassing previous approaches with notable average mAP gains of +3.3%, +2.6%, +1.2%, +1.7% (verb), and +1.4% (noun), respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。