用测试时训练提升视觉模型效率与表达力,大幅降低计算开销。
Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training
- 将视觉序列视为数据集,通过自监督压缩实现线性时间建模。
- ImageNet上达到82.7%准确率,1280×1280分辨率下提速4.72倍。
- 适合追求高效通用视觉主干的工程与研究应用。
高效且富有表现力的视觉表征学习是计算机视觉长期追求的目标。尽管视觉变压器(ViTs)逐步取代传统卷积神经网络(CNNs)成为更可扩展的视觉学习器,但其应用受限于自注意力机制的二次复杂度。为此,我们首次将测试时训练(TTT)引入视觉领域,提出Vision-TTT,将视觉序列视为数据集,以新颖的自监督方式压缩视觉标记序列。结合双数据集策略和基于Conv2d的数据预处理,Vision-TTT有效扩展了原始TTT,实现对二维视觉相关性的建模并具备全局感受野。大量实验表明, exttt{Vittt-T/S/B}在ImageNet分类任务中分别取得77.7%、81.8%、82.7%的Top-1准确率,在下游任务中显著优于对应基线模型。在1280×1280分辨率下, exttt{Vittt-T}相比DeiT-T减少79.4%的浮点运算量,运行速度提升4.72倍,内存占用降低88.9%。这些结果证明Vision-TTT作为下一代通用视觉主干的强大潜力。
原文摘要 · Abstract (English)
Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which treats visual sequences as datasets and compresses the visual token sequences in a novel self-supervised learning manner. By incorporating the dual-dataset strategy and Conv2d-based dataset preprocessing, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that \texttt{Vittt-T/S/B} achieve $77.7\%,81.8\%,82.7\%$ Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At $1280\times1280$ resolution, \texttt{Vittt-T} reduces FLOPs by $79.4\%$ and runs $4.72\times$ faster with $88.9\%$ less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。