用稀疏轨迹识别动作，避免背景干扰，精度超主流模型
TrAction: Action Recognition with Sparse Trajectories
- 用稀疏点轨迹替代密集视频帧，降低计算负担
- 在Something-Something V2上达45%准确率，优于现有方法
- 轨迹特征与外观特征互补，融合后提升显著
当前动作识别模型依赖高内存、高算力的密集RGB视频，常受外观和背景线索干扰，如通过物体或场景而非运动特征判断动作。本文提出一种基于稀疏点轨迹的高效替代输入，设计了一个简单的2.5D Transformer架构，并引入掩码轨迹预训练策略，显著提升下游动作识别准确率。仅使用少量密集视频输入，模型在Something-Something V2上达到45% top-1准确率，在EPIC-Kitchens-100上达54%，且在时间反转敏感性测试中超越V-JEPA。更重要的是，轨迹特征与先进外观特征（如DINOv2、V-JEPA 2）具有互补性，融合后在Something-Something V2上分别提升8.7和1.6个百分点。
原文摘要 · Abstract (English)
Modern action recognition models operate on memory- and compute-intensive dense RGB video volumes and frequently exploit appearance and background shortcuts, for example, predicting actions from objects or scenes instead of characteristic motion. We investigate an efficient alternative input modality that is largely free of such biases by construction: sparse point trajectories. To this end, we develop a simple transformer architecture for 2.5D trajectory-based recognition together with a masked-trajectory pretraining, which we show to substantially improve downstream action recognition accuracy. Despite using only a fraction of the dense RGB input, our method reaches 45% top-1 on Something-Something V2 and 54% on EPIC-Kitchens-100, and surpasses V-JEPA on time-reversal sensitivity. More importantly, we find trajectory features to be complementary to state-of-the-art appearance-based features. Fusing our pretrained model with DINOv2 and V-JEPA 2 improves top-1 accuracy on Something-Something V2 by 8.7 and 1.6 points, respectively. Code: https://github.com/ecker-lab/TrAction
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。