模仿人类视觉，用全局凝视与局部聚焦提升视频理解能力
Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer
- 设计双路径结构：全局凝视捕整体动态，局部聚焦补细节信息
- 在Kinetics-400等三个数据集上达到顶尖性能，超越现有方法
- 适合关注视频动作理解、高效注意力机制的研究者参考
Transformer在视觉任务中取得显著进展。为平衡视频任务的计算效率，现有方法多采用分解或窗口化自注意力，但这类方法割裂了视频中感兴趣区域间的时空关联，限制了对运动和长距离依赖的捕捉能力。本文受人类视觉系统启发，认为时间和空间信息的重要性随时间尺度变化，注意力呈稀疏时序分布，表现为凝视与聚焦行为。我们提出一种双路径网络——整体凝视与精细聚焦（OG-ReG）Transformer：凝视路径提取粗粒度时空信息，聚焦路径则补充局部细节。该模型在Kinetics-400、Something-Something v2和Diving-48数据集上均达到当前最优表现，验证了其有效性。代码将开源于https://github.com/linuxsino/OG-ReG。
原文摘要 · Abstract (English)
Recently, Transformer has made significant progress in various vision tasks. To balance computation and efficiency in video tasks, recent works heavily rely on factorized or window-based self-attention. However, these approaches split spatiotemporal correlations between regions of interest in videos, limiting the models' ability to capture motion and long-range dependencies. In this paper, we argue that, similar to the human visual system, the importance of temporal and spatial information varies across different time scales, and attention is allocated sparsely over time through glance and gaze behavior. Is equal consideration of time and space crucial for success in video tasks? Motivated by this understanding, we propose a dual-path network called the Overall Glance and Refined Gaze (OG-ReG) Transformer. The Glance path extracts coarse-grained overall spatiotemporal information, while the Gaze path supplements the Glance path by providing local details. Our model achieves state-of-the-art results on the Kinetics-400, Something-Something v2, and Diving-48, demonstrating its competitive performance. The code will be available at https://github.com/linuxsino/OG-ReG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。