arXiv:2512.01643cs.CV2025-12被引 8

提出视觉领域测试时训练新模型,实现线性复杂度下的高效性能。

ViT$^3$: Unlocking Test-Time Training in Vision

  • 将注意力机制改造成测试时在线学习,构建轻量内层模型。
  • 在图像分类、检测等任务上超越同类线性模型,接近Transformer性能。
  • 提供6条设计原则,为视觉TTT研究提供实用指导。

测试时训练(TTT)近期成为高效序列建模的有前景方向。TTT将注意力操作重构为在线学习问题,在测试时从键值对中构建紧凑内层模型,开辟了丰富灵活的设计空间,同时实现线性计算复杂度。然而,设计高效的视觉TTT仍具挑战:内层模块和内层训练的关键选择缺乏全面理解与实践指南。本文通过系统实证研究,针对视觉序列建模的TTT设计展开分析,提炼出六条实用洞见,确立有效视觉TTT的设计原则,并指引未来改进路径。由此形成纯TTT架构——视觉测试时训练模型(ViT$^3$),具备线性复杂度与可并行计算特性。我们在图像分类、生成、目标检测与语义分割等多样化视觉任务上评估了ViT$^3$,结果表明其性能持续优于或媲美先进线性复杂度模型(如Mamba与线性注意力变体),并显著缩小与高度优化的视觉Transformer之间的差距。我们希望本研究及ViT$^3$基线能推动视觉TTT模型的后续研究。代码开源:github.com/LeapLabTHU/ViTTT。

原文摘要 · Abstract (English)

Test-Time Training (TTT) has recently emerged as a promising direction for efficient sequence modeling. TTT reformulates attention operation as an online learning problem, constructing a compact inner model from key-value pairs at test time. This reformulation opens a rich and flexible design space while achieving linear computational complexity. However, crafting a powerful visual TTT design remains challenging: fundamental choices for the inner module and inner training lack comprehensive understanding and practical guidelines. To bridge this critical gap, in this paper, we present a systematic empirical study of TTT designs for visual sequence modeling. From a series of experiments and analyses, we distill six practical insights that establish design principles for effective visual TTT and illuminate paths for future improvement. These findings culminate in the Vision Test-Time Training (ViT$^3$) model, a pure TTT architecture that achieves linear complexity and parallelizable computation. We evaluate ViT$^3$ across diverse visual tasks, including image classification, image generation, object detection, and semantic segmentation. Results show that ViT$^3$ consistently matches or outperforms advanced linear-complexity models (e.g., Mamba and linear attention variants) and effectively narrows the gap to highly optimized vision Transformers. We hope this study and the ViT$^3$ baseline can facilitate future work on visual TTT models. Code: github.com/LeapLabTHU/ViTTT.

测试时训练视觉模型线性复杂度Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。