arXiv:2512.19941cs.CVcs.AI2025-12被引 14

发现视觉Transformer深层计算可压缩为少量重复块,提升可解释性。

Block-Recurrent Dynamics in Vision Transformers

  • 提出块循环假说:深度计算可用少数块反复迭代实现。
  • 仅用2个块复现DINOv2 96%的线性探针准确率,等效运行时间。
  • 揭示分类方向收敛、令牌动态差异与低秩更新等可解释规律。

随着视觉Transformer(ViT)成为主流视觉骨干网络,其计算机制的理解至关重要。尽管架构暗示动态结构,但尚无统一框架将深度视为明确演化过程。本文提出块循环假说(BRH),认为训练后的ViT具有块循环结构,即原L个块的计算可由仅k≪L个不同块反复应用准确重构。在多种ViT中,层间表征相似性矩阵显示少数组连续阶段。为验证这些阶段是否反映可复用计算,我们训练了预训练ViT的块循环替代模型——块结构变换器循环近似器(Raptor)。小规模实验表明,随机深度和训练促进循环结构,并与拟合Raptor的能力相关。进一步通过实证证明了BRH:训练一个Raptor模型,在仅2个块且等效运行时间内恢复了DINOv2 ImageNet-1k线性探针96%的准确率。基于该假说,我们发展出动力学可解释性研究框架,发现:i)分类方向收敛至类相关的角基域,微小扰动下具自校正轨迹;ii)令牌特定动态:cls token在后期出现剧烈重定向,而patch token在晚期呈现强平均方向一致性;iii)晚期深度出现低秩更新,与收敛至低维吸引子一致。总体而言,沿ViT深度涌现出紧凑的循环程序,指向一种低复杂度的规范解,使这些模型可通过严谨的动力系统分析进行研究。

原文摘要 · Abstract (English)

As Vision Transformers (ViTs) become standard vision backbones, a mechanistic account of their computational phenomenology is essential. Despite architectural cues that hint at dynamical structure, there is no settled framework that interprets Transformer depth as a well-characterized flow. In this work, we introduce the Block-Recurrent Hypothesis (BRH), arguing that trained ViTs admit a block-recurrent depth structure such that the computation of the original $L$ blocks can be accurately rewritten using only $k \ll L$ distinct blocks applied recurrently. Across diverse ViTs, between-layer representational similarity matrices suggest few contiguous phases. To determine whether these phases reflect genuinely reusable computation, we train block-recurrent surrogates of pretrained ViTs: Recurrent Approximations to Phase-structured TransfORmers (Raptor). In small-scale, we demonstrate that stochastic depth and training promote recurrent structure and subsequently correlate with our ability to accurately fit Raptor. We then provide an empirical existence proof for BRH by training a Raptor model to recover $96\%$ of DINOv2 ImageNet-1k linear probe accuracy in only 2 blocks at equivalent runtime. Finally, we leverage our hypothesis to develop a program of Dynamical Interpretability. We find i) directional convergence into class-dependent angular basins with self-correcting trajectories under small perturbations, ii) token-specific dynamics, where cls executes sharp late reorientations while patch tokens exhibit strong late-stage coherence toward their mean direction, and iii) a collapse to low rank updates in late depth, consistent with convergence to low-dimensional attractors. Altogether, we find a compact recurrent program emerges along ViT depth, pointing to a low-complexity normative solution that enables these models to be studied through principled dynamical systems analysis.

视觉Transformer动力系统可解释性块循环

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。