用跨模态注意力实现视频动作的时空音频统一理解
CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition
- 设计双流结构，通过交叉注意力让空间与时间专家协同预测
- 在多个数据集上均实现平衡的性能，最高准确率达81.3%
- 适合需要多模态视频理解的场景，如智能监控与内容分析
我们提出一种基于Transformer的全视频理解方法CA^2ST，旨在解决现有模型在时空理解上的不平衡问题。核心是Cross-Attention in Space and Time（CAST）架构，仅使用RGB输入，在每一层通过瓶颈交叉注意力（B-CA）实现空间与时间专家的信息交换与协同决策。为进一步实现全景理解，引入音频专家形成跨模态的CAVA。在EPIC-KITCHENS-100、Something-Something-V2和Kinetics-400等基准上，CAST表现均衡；在UCF-101、VGG-Sound、KineticsSound和EPIC-SOUNDS上，CAVA在多个数据集上达到81.3%的最高准确率。结果表明，B-CA模块能有效促进多专家间信息融合。最终，CA^2ST通过空间、时间与音频专家的交叉注意力机制，实现更全面、均衡的视频理解。
原文摘要 · Abstract (English)
We propose Cross-Attention in Audio, Space, and Time (CA^2ST), a transformer-based method for holistic video recognition. Recognizing actions in videos requires both spatial and temporal understanding, yet most existing models lack a balanced spatio-temporal understanding of videos. To address this, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), using only RGB input. In each layer of CAST, Bottleneck Cross-Attention (B-CA) enables spatial and temporal experts to exchange information and make synergistic predictions. For holistic video understanding, we extend CAST by integrating an audio expert, forming Cross-Attention in Visual and Audio (CAVA). We validate the CAST on benchmarks with different characteristics, EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400, consistently showing balanced performance. We also validate the CAVA on audio-visual action recognition benchmarks, including UCF-101, VGG-Sound, KineticsSound, and EPIC-SOUNDS. With a favorable performance of CAVA across these datasets, we demonstrate the effective information exchange among multiple experts within the B-CA module. In summary, CA^2ST combines CAST and CAVA by employing spatial, temporal, and audio experts through cross-attention, achieving balanced and holistic video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。