用Mamba结构实现高效长时序动作预测，兼顾精度与速度。
MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipation
- 采用Mamba架构，线性复杂度建模超长视频时序依赖。
- 在3个数据集上达到当前最佳性能，计算开销大幅降低。
- 适合需要快速、精准长期动作预判的智能系统应用。
长期密集动作预测极具挑战性，需基于视频观测预测数分钟后的动作及其持续时间。为建模未来结果的不确定性，随机模型会为同一观测生成多个可能的未来动作序列。近期工作进一步提出统一建模观察帧的过去与未来动作，以提升预测效果。然而，这种联合建模依赖长时程依赖关系，而此前方法受限于有限或稀疏的感受野，难以捕捉远距离事件关联。为此，我们提出新型MANTA（MAmba for ANTicipation）网络，可在极长序列中实现有效长时序建模，同时保持序列长度的线性复杂度。实验表明，该方法在Breakfast、50Salads、Assembly101三个数据集上均取得当前最优结果，并显著提升计算与内存效率。代码已开源：https://github.com/olga-zats/DIFF_MANTA。
原文摘要 · Abstract (English)
Long-term dense action anticipation is very challenging since it requires predicting actions and their durations several minutes into the future based on provided video observations. To model the uncertainty of future outcomes, stochastic models predict several potential future action sequences for the same observation. Recent work has further proposed to incorporate uncertainty modelling for observed frames by simultaneously predicting per-frame past and future actions in a unified manner. While such joint modelling of actions is beneficial, it requires long-range temporal capabilities to connect events across distant past and future time points. However, the previous work struggles to achieve such a long-range understanding due to its limited and/or sparse receptive field. To alleviate this issue, we propose a novel MANTA (MAmba for ANTicipation) network. Our model enables effective long-term temporal modelling even for very long sequences while maintaining linear complexity in sequence length. We demonstrate that our approach achieves state-of-the-art results on three datasets - Breakfast, 50Salads, and Assembly101 - while also significantly improving computational and memory efficiency. Our code is available at https://github.com/olga-zats/DIFF_MANTA .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。