新模型精准定位动作起止时间，尤其擅长处理模糊边界。
TBT-Former: Learning Temporal Boundary Distributions for Action Localization
- 用改进的Transformer和多尺度融合网络增强时序特征提取能力。
- 提出分布回归头，在THUMOS14上达到86.3% mAP，领先现有方法。
- 适合需要高精度动作定位的视频分析任务，如体育动作识别。
时序动作定位（TAL）是视频理解中的核心挑战，旨在识别未剪辑视频中所有动作实例的起始时间、结束时间及类别。尽管近期基于Transformer的单阶段无锚点模型（如ActionFormer）已取得显著进展，但仍面临两个关键问题：动作边界模糊时的精确定位困难，以及多尺度上下文信息的有效融合不足。本文提出时序边界变换器（TBT-Former），在ActionFormer基础上实现三项改进：(1) 采用更高容量的缩放式Transformer主干网络，增加注意力头数量与MLP维度，提升时序特征表达能力；(2) 设计包含自顶向下路径与横向连接的跨尺度特征金字塔网络（FPN），实现高层语义与低层时序细节的丰富融合；(3) 引入新颖的边界分布回归头，借鉴广义焦点损失（GFL）思想，将边界回归转化为概率分布学习，显式建模边界不确定性。在具有挑战性的THUMOS14与EPIC-Kitchens 100数据集上，TBT-Former显著超越前人基准，取得86.3% mAP（THUMOS14）与79.2% mAP（EPIC-Kitchens 100）的性能，并在大型数据集ActivityNet-1.3上保持竞争力。代码已开源。
原文摘要 · Abstract (English)
Temporal Action Localization (TAL) remains a fundamental challenge in video understanding, aiming to identify the start time, end time, and category of all action instances within untrimmed videos. While recent single-stage, anchor-free models like ActionFormer have set a high standard by leveraging Transformers for temporal reasoning, they often struggle with two persistent issues: the precise localization of actions with ambiguous or "fuzzy" temporal boundaries and the effective fusion of multi-scale contextual information. In this paper, we introduce the Temporal Boundary Transformer (TBT-Former), a new architecture that directly addresses these limitations. TBT-Former enhances the strong ActionFormer baseline with three core contributions: (1) a higher-capacity scaled Transformer backbone with an increased number of attention heads and an expanded Multi-Layer Perceptron (MLP) dimension for more powerful temporal feature extraction; (2) a cross-scale feature pyramid network (FPN) that integrates a top-down pathway with lateral connections, enabling richer fusion of high-level semantics and low-level temporal details; and (3) a novel boundary distribution regression head. Inspired by the principles of Generalized Focal Loss (GFL), this new head recasts the challenging task of boundary regression as a more flexible probability distribution learning problem, allowing the model to explicitly represent and reason about boundary uncertainty. Within the paradigm of Transformer-based architectures, TBT-Former advances the formidable benchmark set by its predecessors, establishing a new level of performance on the highly competitive THUMOS14 and EPIC-Kitchens 100 datasets, while remaining competitive on the large-scale ActivityNet-1.3. Our code is available at https://github.com/aaivu/In21-S7-CS4681-AML-Research-Projects/tree/main/projects/210536K-Multi-Modal-Learning_Video-Understanding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。