发现视觉Transformer深度可近似为线性动态系统。
Dynamic Mode Decomposition along Depth in Vision Transformers
- 用动态模态分解方法提取跨层的统一线性算子
- 短跨度预测误差小于0.02余弦相似度，能还原中间状态
- 早期层更易线性化，适合压缩与加速
近期研究表明，连续的视觉Transformer（ViT）模块可被线性映射替代，并呈现出周期性计算模式。本文探讨这些现象是否一致：视觉Transformer的深度是否实现了近似自洽的线性动态？我们采用动态模态分解（DMD）方法，从连续隐藏状态对中拟合单一算子 $K$，并用 $K^p$ 预测 $p$ 步后状态。在四个预训练的DINO ViTs上，研究了正则化、秩与校准预算对稳定拟合的影响。对于短跨度（$p \≤ 4$），$K^p$ 在DINOv3-H/16+上与无约束终点映射的余弦相似度达0.02以内，且能恢复跳过的各块中间激活。早期截断时，拟合算子秩远小于特征维度 $d$，仅需少量校准数据；所有标记中， exttt{cls} 最适合线性化，且该特性随深度单调下降。然而，这种局部保真度无法传递至下游任务：最终隐藏状态下，恒等基线表现已具竞争力。
原文摘要 · Abstract (English)
Recent work has shown that contiguous vision transformer (ViT) blocks (a) can be replaced by a linear map and (b) organize into recurrent phases of computation. We ask whether these observations coincide: does ViT depth implement approximately \textit{autonomous linear} dynamics, admitting a single operator $K$ applied recurrently across a contiguous span? We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs and predicts $p$ steps ahead via $K^p$. On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting. For short spans ($p \leq 4$), $K^p$ tracks an unconstrained endpoint map to within $0.02$ cosine similarity on DINOv3-H/16+, while also recovering intermediate activations at each skipped block. At early cut starts, the fitted operators compress to rank $\ll d$ with minimal calibration data, and across tokens, \texttt{cls} is most amenable to linearization; both properties decay monotonically with depth. Yet this local fidelity does not transfer downstream. At the final hidden state, after propagating through the remaining blocks, an identity baseline becomes competitive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。