融合音视频特征与加权框融合，提升动作定位精度。
The Solution for Temporal Action Localisation Task of Perception Test Challenge 2024
- 用重叠标签扩增数据，增强模型泛化能力。
- 多模态与单模态模型融合预测，取得0.5498得分。
- 适合关注视频动作定位与多模态融合的研究者。
本文提出一种用于时间动作定位（Temporal Action Localisation, TAL）的方法，旨在识别视频序列中特定时间段内的动作。通过在Something-SomethingV2数据集上使用重叠标签进行数据增强，扩展训练数据，提升模型对各类动作的泛化能力。特征提取采用先进模型：视频特征使用UMT和VideoMAEv2，音频特征使用BEATs和CAV-MAE。我们训练了多模态（视频+音频）与单模态（仅视频）模型，并通过加权框融合（Weighted Box Fusion, WBF）方法合并预测结果，实现鲁棒的动作定位。该方法在感知测试挑战赛2024中取得0.5498的最终得分，排名第一。
原文摘要 · Abstract (English)
This report presents our method for Temporal Action Localisation (TAL), which focuses on identifying and classifying actions within specific time intervals throughout a video sequence. We employ a data augmentation technique by expanding the training dataset using overlapping labels from the Something-SomethingV2 dataset, enhancing the model's ability to generalize across various action classes. For feature extraction, we utilize state-of-the-art models, including UMT, VideoMAEv2 for video features, and BEATs and CAV-MAE for audio features. Our approach involves training both multimodal (video and audio) and unimodal (video only) models, followed by combining their predictions using the Weighted Box Fusion (WBF) method. This fusion strategy ensures robust action localisation. our overall approach achieves a score of 0.5498, securing first place in the competition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。