用多尺度时序Mamba模型提升长视频动作检测精度
MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videos
- 引入扩张状态空间模块，捕捉跨尺度时间特征
- 仅1700万参数即在TSU和Charades上达顶尖性能
- 适合处理长时序、重叠动作的视频理解任务
未剪辑视频中的时序动作检测（TAD）面临重大挑战，尤其针对日常生活活动（ADL），要求模型能（1）处理长时视频，（2）捕捉动作的时间变化，（3）同时检测密集重叠的动作。现有基于CNN和Transformer的方法难以兼顾细粒度细节与长程结构。基于状态空间模型（SSM）的Mamba具备强大的长程建模能力，但直接应用于TAD会破坏细粒度时间结构，且无法应对TAD固有挑战。为此，我们提出多尺度时序Mamba（MS-Temba），通过引入扩张状态空间模块扩展Mamba用于TAD。每个Temba模块包含扩张SSM及新设计的辅助损失，可学习跨时间尺度的判别性表征。轻量级多尺度Mamba融合器通过基于SSM的聚合方式统一多尺度特征，实现精准的动作边界定位。仅需1700万参数，MS-Temba在密集标注的ADL基准数据集TSU与Charades上达到当前最优表现，并进一步泛化至长视频摘要任务，在TVSum与SumMe上刷新纪录。
原文摘要 · Abstract (English)
Temporal Action Detection (TAD) in untrimmed videos poses significant challenges, particularly for Activities of Daily Living (ADL) requiring models to (1) process long-duration videos, (2) capture temporal variations in actions, and (3) simultaneously detect dense overlapping actions. Existing CNN and Transformer-based approaches, struggle to jointly capture fine-grained detail and long-range structure at scale. State-space Model (SSM) based Mamba offers powerful long-range modeling, but naive application to TAD collapses fine-grained temporal structure and fails to account for the challenges inherent to TAD. To this end, we propose Multi-Scale Temporal Mamba (MS-Temba), which extends Mamba to TAD with newly introduced dilated SSMs. Each Temba block, comprising dilated SSMs coupled with our proposed additional losses, enables the learning of discriminative representations across temporal scales. A lightweight Multi-scale Mamba Fuser then unifies these multi-scale features via SSM-based aggregation, yielding precise action-boundary localization. With only 17M parameters, MS-Temba achieves state-of-the-art performance on densely labeled ADL benchmarks TSU & Charades, and further generalizes to long-form video summarization, setting new state-of-the-art results on TVSum & SumMe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。