将ViT重构成微分方程系统，参数少10倍仍保持高精度。
ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equation
- 把ViT看作常微分方程，实现连续动态建模。
- 在CIFAR-10/100上参数减少一个数量级，性能媲美原始ViT。
- 提出即插即用师生框架，性能提升超10%。
近年来，大规模模型在计算机视觉任务中表现优异，但其计算开销大、结构复杂，难以解释。多数架构依赖Transformer中的注意力机制。基于残差网络与常微分方程（ODE）的联系，本文提出ODE-ViT，将视觉Transformer重构为满足良好适定性和稳定性的ODE系统。在CIFAR-10和CIFAR-100上的实验表明，ODE-ViT在参数量减少一个数量级的情况下，仍能实现稳定、可解释且具有竞争力的分类性能，优于此前基于ODE的Transformer方法。我们进一步提出一种即插即用的教师-学生框架，将离散ViT的中间表示作为ODE解，指导ODE-ViT的连续轨迹学习。该策略使性能相比从头训练的自由ODE-ViT提升超过10%。
原文摘要 · Abstract (English)
In recent years, increasingly large models have achieved outstanding performance across CV tasks. However, these models demand substantial computational resources and storage, and their growing complexity limits our understanding of how they make decisions. Most of these architectures rely on the attention mechanism within Transformer-based designs. Building upon the connection between residual neural networks and ordinary differential equations (ODEs), we introduce ODE-ViT, a Vision Transformer reformulated as an ODE system that satisfies the conditions for well-posed and stable dynamics. Experiments on CIFAR-10 and CIFAR-100 demonstrate that ODE-ViT achieves stable, interpretable, and competitive performance with up to one order of magnitude fewer parameters, surpassing prior ODE-based Transformer approaches in classification tasks. We further propose a plug-and-play teacher-student framework in which a discrete ViT guides the continuous trajectory of ODE-ViT by treating the intermediate representations of the teacher as solutions of the ODE. This strategy improves performance by more than 10% compared to training a free ODE-ViT from scratch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。