发现视觉Transformer深度增加反而性能下降的内在机制
Mechanisms of Non-Monotonic Scaling in Vision Transformers
- 提出三阶段演化模式：悬崖-平台-爬升，解释深层模型性能变化
- 发现[CLS] token被弱化，特征由局部块令牌共同决定
- 提出信息打乱指数，可诊断模型状态并指导未来设计
更深的视觉Transformer（ViT）在ImageNet上表现常不如浅层模型，挑战了常规的扩展假设。通过对ViT-S、ViT-B和ViT-L的系统性实证分析，我们发现表征演化遵循一致的三阶段‘悬崖-平台-爬升’模式。性能提升与[CLS] token逐渐被弱化相关，其原设计为全局聚合中心，而实际演变为块令牌间的分布式共识。通过信息打乱指数量化信息混合模式，发现ViT-L中信息-任务权衡约比ViT-B晚10层出现，且这些额外层数主要导致信息扩散增强而非任务性能提升。结果表明，在此范围内，模型更受益于精心校准的深度以实现清晰相变，而非单纯增加参数量。该指数可用于现有模型诊断，并为未来架构提供设计目标。
原文摘要 · Abstract (English)
Deeper Vision Transformers often perform worse than shallower ones, which challenges common scaling assumptions. Through a systematic empirical analysis of ViT-S, ViT-B, and ViT-L on ImageNet, we identify a consistent three-phase Cliff-Plateau-Climb pattern that governs how representations evolve with depth. We observe that better performance is associated with progressive marginalization of the [CLS] token, originally designed as a global aggregation hub, in favor of distributed consensus among patch tokens. We quantify patterns of information mixing with an Information Scrambling Index, and show that in ViT-L the information-task tradeoff emerges roughly 10 layers later than in ViT-B, and that these additional layers correlate with increased information diffusion rather than improved task performance. Taken together, these results suggest that transformer architectures in this regime may benefit more from carefully calibrated depth that executes clean phase transitions than from simply increasing parameter count. The Information Scrambling Index provides a useful diagnostic for existing models and suggests a potential design target for future architectures. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。