arXiv:2510.16209cs.CV2025-10被引 1

让视频状态空间模型灵活适应不同长短和分辨率,提升动作识别效果

StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales

  • 训练时随机采样不同时空分辨率视频,动态插值权重以适配任意尺度
  • 在短动作(UCF-101)和长动作(COIN)数据集上性能领先基线达28%
  • 适合需要跨尺度视频理解的场景,如从短视频到长视频的动作识别

状态空间模型(SSMs)因其线性复杂度和隐状态循环特性,在建模长序列方面优于注意力机制。然而现有视频理解训练方法针对Transformer设计,未能充分发挥SSM优势。当前模型常固定分辨率与长度训练,导致在未见时空尺度下性能下降,即存在时空不灵活问题。为此,我们提出柔性训练方法:训练中动态采样多时空分辨率视频,并通过权重插值实现任意尺度自适应。由此构建的模型StretchySnake具备时空灵活性,可无缝处理从短而精细的片段到长而复杂的活动。我们在五种柔性训练变体中对比并筛选最优策略。在短动作(UCF-101、HMDB-51)与长动作(COIN、Breakfast)基准上,其性能较Transformer与传统SSM基线最高提升28%,对细粒度动作(SSV2、Diving-48)也表现优异。该方法为视频SSM提供简单有效的训练方案,显著提升鲁棒性、分辨率无关性与效率。

原文摘要 · Abstract (English)

State space models (SSMs) have emerged as a competitive alternative to transformers in various tasks. Their linear complexity and hidden-state recurrence make them particularly attractive for modeling long sequences, whereas attention becomes quadratically expensive. However, current training methods for video understanding are tailored towards transformers and fail to fully leverage the unique attributes of SSMs. For example, video models are often trained at a fixed resolution and video length to balance the quadratic scaling of attention cost against performance. Consequently, these models suffer from degraded performance when evaluated on videos with spatial and temporal resolutions unseen during training; a property we call spatio-temporal inflexibility. In the context of action recognition, this severely limits a model's ability to retain performance across both short- and long-form videos. Therefore, we propose a flexible training method that leverages and improves the inherent adaptability of SSMs. Our method samples videos at varying temporal and spatial resolutions during training and dynamically interpolates model weights to accommodate any spatio-temporal scale. This instills our SSM, which we call StretchySnake, with spatio-temporal flexibility and enables it to seamlessly handle videos ranging from short, fine-grained clips to long, complex activities. We introduce and compare five different variants of flexible training, and identify the most effective strategy for video SSMs. On short-action (UCF-101, HMDB-51) and long-action (COIN, Breakfast) benchmarks, StretchySnake outperforms transformer and SSM baselines alike by up to 28%, with strong adaptability to fine-grained actions (SSV2, Diving-48). Therefore, our method provides a simple drop-in training recipe that makes video SSMs more robust, resolution-agnostic, and efficient across diverse action recognition scenarios.

视频理解状态空间模型动作识别灵活训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。