改进时间动作检测的编码解码结构，提升长时序建模能力
DiGIT: Multi-Dilated Gated Encoder and Central-Adjacent Region Integrated Decoder for Temporal Action Detection Transformer
- 采用多膨胀门控编码器减少多尺度特征冗余
- 在THUMOS14等数据集上达到最新最好性能
- 适合需要精准时序定位的动作识别研究者
本文针对基于查询的时间动作检测（TAD）模型在直接沿用目标检测架构时存在的关键缺陷展开研究。尽管现有模型表现良好，但仍难以充分应对TAD特有的挑战，如多尺度特征冗余和有限的时序上下文捕捉能力。为此，我们提出一种面向时间动作检测变压器的多膨胀门控编码器与中心邻近区域集成解码器（DiGIT）。新编码器替代原有包含多尺度可变形注意力与前馈网络的结构，有效降低多层级特征带来的冗余信息，同时保持对细粒度与长距离时序信息的捕捉能力。此外，我们设计了中心邻近区域集成解码器，通过更全面的采样策略优化可变形交叉注意力，以提取关键信息。大量实验表明，DiGIT在THUMOS14、ActivityNet v1.3和HACS-Segment上均达到当前最优性能。代码已公开于：https://github.com/Dotori-HJ/DiGIT
原文摘要 · Abstract (English)
In this paper, we examine a key limitation in query-based detectors for temporal action detection (TAD), which arises from their direct adaptation of originally designed architectures for object detection. Despite the effectiveness of the existing models, they struggle to fully address the unique challenges of TAD, such as the redundancy in multi-scale features and the limited ability to capture sufficient temporal context. To address these issues, we propose a multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer (DiGIT). Our approach replaces the existing encoder that consists of multi-scale deformable attention and feedforward network with our multi-dilated gated encoder. Our proposed encoder reduces the redundant information caused by multi-level features while maintaining the ability to capture fine-grained and long-range temporal information. Furthermore, we introduce a central-adjacent region integrated decoder that leverages a more comprehensive sampling strategy for deformable cross-attention to capture the essential information. Extensive experiments demonstrate that DiGIT achieves state-of-the-art performance on THUMOS14, ActivityNet v1.3, and HACS-Segment. Code is available at: https://github.com/Dotori-HJ/DiGIT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。