ViT在中间层自发出现信息压缩，是数据驱动的特征提炼机制。
The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers
- 通过分析不同数据集的编码维度变化，发现瓶颈深度随语义抽象需求而变。
- 纹理类数据保持高秩表示，物体类数据在中间层抑制高频信息。
- 揭示了ViT无需显式结构就能自适应学习特征压缩，适合视觉任务研究者。
视觉变换器（ViTs）缺乏卷积神经网络（CNNs）的层次化归纳偏置，理论上可在各层维持高维表示。然而近期观察表明，ViTs常在中间层出现信息压缩，形成“U型”熵曲线。本文通过分析DINO训练的ViTs在不同组成复杂度数据集（UC Merced、Tiny ImageNet、CIFAR-100）上的逐层有效编码维度（EED），发现瓶颈深度与任务所需语义抽象程度强相关。结果表明，纹理密集型数据保持高秩表示，而以物体为中心的数据促使网络在中间层压制高频信息，主动“学习”出瓶颈以分离语义特征，证明该现象非架构缺陷，而是数据驱动的适应性行为。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) lack the hierarchical inductive biases inherent to Convolutional Neural Networks (CNNs), theoretically allowing them to maintain high-dimensional representations throughout all layers. However, recent observations suggest ViTs often spontaneously manifest a "U-shaped" entropy profile-compressing information in middle layers before expanding it for the final classification. In this work, we demonstrate that this "Inductive Bottleneck" is not an architectural artifact, but a data-dependent adaptation. By analyzing the layer-wise Effective Encoding Dimension (EED) of DINO-trained ViTs across datasets of varying compositional complexity (UC Merced, Tiny ImageNet, and CIFAR-100), we show that the depth of the bottleneck correlates strongly with the semantic abstraction required by the task. We find that while texture-heavy datasets preserve high-rank representations throughout, object-centric datasets drive the network to dampen high-frequency information in middle layers, effectively "learning" a bottleneck to isolate semantic features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。