发现视觉Transformer深层计算可压缩为少量重复块,提升可解释性。
Block-Recurrent Dynamics in Vision Transformers
- 提出块循环假说:深度计算可用少数块反复迭代实现。
- 仅用2个块复现DINOv2 96%的线性探针准确率,等效运行时间。
- 揭示分类方向收敛、令牌动态差异与低秩更新等可解释规律。
随着视觉Transformer(ViT)成为主流视觉骨干网络,其计算机制的理解至关重要。尽管架构暗示动态结构,但尚无统一框架将深度视为明确演化过程。本文提出块循环假说(BRH),认为训练后的ViT具有块循环结构,即原L个块的计算可由仅k≪L个不同块反复应用准确重构。在多种ViT中,层间表征相似性矩阵显示少数组连续阶段。为验证这些阶段是否反映可复用计算,我们训练了预训练ViT的块循环替代模型——块结构变换器循环近似器(Raptor)。小规模实验表明,随机深度和训练促进循环结构,并与拟合Raptor的能力相关。进一步通过实证证明了BRH:训练一个Raptor模型,在仅2个块且等效运行时间内恢复了DINOv2 ImageNet-1k线性探针96%的准确率。基于该假说,我们发展出动力学可解释性研究框架,发现:i)分类方向收敛至类相关的角基域,微小扰动下具自校正轨迹;ii)令牌特定动态:cls token在后期出现剧烈重定向,而patch token在晚期呈现强平均方向一致性;iii)晚期深度出现低秩更新,与收敛至低维吸引子一致。总体而言,沿ViT深度涌现出紧凑的循环程序,指向一种低复杂度的规范解,使这些模型可通过严谨的动力系统分析进行研究。
原文摘要 · Abstract (English)
As Vision Transformers (ViTs) become standard vision backbones, a mechanistic account of their computational phenomenology is essential. Despite architectural cues that hint at dynamical structure, there is no settled framework that interprets Transformer depth as a well-characterized flow. In this work, we introduce the Block-Recurrent Hypothesis (BRH), arguing that trained ViTs admit a block-recurrent depth structure such that the computation of the original $L$ blocks can be accurately rewritten using only $k \ll L$ distinct blocks applied recurrently. Across diverse ViTs, between-layer representational similarity matrices suggest few contiguous phases. To determine whether these phases reflect genuinely reusable computation, we train block-recurrent surrogates of pretrained ViTs: Recurrent Approximations to Phase-structured TransfORmers (Raptor). In small-scale, we demonstrate that stochastic depth and training promote recurrent structure and subsequently correlate with our ability to accurately fit Raptor. We then provide an empirical existence proof for BRH by training a Raptor model to recover $96\%$ of DINOv2 ImageNet-1k linear probe accuracy in only 2 blocks at equivalent runtime. Finally, we leverage our hypothesis to develop a program of Dynamical Interpretability. We find i) directional convergence into class-dependent angular basins with self-correcting trajectories under small perturbations, ii) token-specific dynamics, where cls executes sharp late reorientations while patch tokens exhibit strong late-stage coherence toward their mean direction, and iii) a collapse to low rank updates in late depth, consistent with convergence to low-dimensional attractors. Altogether, we find a compact recurrent program emerges along ViT depth, pointing to a low-complexity normative solution that enables these models to be studied through principled dynamical systems analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。