分阶段用Transformer精准定位视频中的动作起止时间。
Hierarchical Multi-Stage Transformer Architecture for Context-Aware Temporal Action Localization
- 分三阶段：候选段生成、动作分类、边界精修，每阶段专用Transformer模块。
- 在THUMOS14等3个数据集上分别提升2.8%、1.2%、4.8%，超越现有方法。
- 适合研究视频理解、动作定位或想用分阶段Transformer的开发者。
受视觉识别与目标检测中Transformer及多阶段架构成功启发，本文深入探索了多阶段架构下Transformer丰富的时空特性，用于时序动作定位（TAL）任务。提出一种分层多阶段Transformer架构PCL-Former，每个子任务由专用Transformer模块处理，并配备特定损失函数：Proposal-Former从未剪辑视频中识别可能包含动作的候选段，Classification-Former对候选段内动作类别进行分类，Localization-Former精确预测动作实例的时间边界（起始与结束）。在THUMOS-14、ActivityNet-1.3和HACS Segments三个挑战性基准数据集上进行大量实验，并开展详尽消融实验验证各模块影响。定量结果证实PCL-Former有效性，在三个数据集上分别优于当前最优方法2.8%、1.2%和4.8%。
原文摘要 · Abstract (English)
Inspired by the recent success of transformers and multi-stage architectures in video recognition and object detection domains. We thoroughly explore the rich spatio-temporal properties of transformers within a multi-stage architecture paradigm for the temporal action localization (TAL) task. This exploration led to the development of a hierarchical multi-stage transformer architecture called PCL-Former, where each subtask is handled by a dedicated transformer module with a specialized loss function. Specifically, the Proposal-Former identifies candidate segments in an untrimmed video that may contain actions, the Classification-Former classifies the action categories within those segments, and the Localization-Former precisely predicts the temporal boundaries (i.e., start and end) of the action instances. To evaluate the performance of our method, we have conducted extensive experiments on three challenging benchmark datasets: THUMOS-14, ActivityNet-1.3, and HACS Segments. We also conducted detailed ablation experiments to assess the impact of each individual module of our PCL-Former. The obtained quantitative results validate the effectiveness of the proposed PCL-Former, outperforming state-of-the-art TAL approaches by 2.8%, 1.2%, and 4.8% on THUMOS14, ActivityNet-1.3, and HACS datasets, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。