通过控制视觉Transformer中注意力的信息流，实现从独立处理到全局交互的可控切换。
From independent patches to coordinated attention: Controlling information flow in vision transformers
- 在残差流上插入变分信息瓶颈，显式量化注意力传递的信息量
- 在ImageNet-100上观察到分类行为随信息流变化的连续谱
- 首次分析了早期注意力头如何从局部块生成全局视觉表征
我们使视觉Transformer中注意力传递的信息成为可显式度量的量。通过在所有注意力写入残差流时插入变分信息瓶颈（无需其他架构修改），我们训练出具有显式信息成本的模型，获得从独立块处理到完全表达的全局注意力之间的可控谱系。在ImageNet-100上，我们刻画了分类行为与信息路由随该谱系的变化，并通过分析最早传递信息的注意力头，初步揭示了全局视觉表征如何由局部块处理演化而来。通过引导学习偏向内部通信受限的解，该方法使模型更易进行机制分析和控制。
原文摘要 · Abstract (English)
We make the information transmitted by attention an explicit, measurable quantity in vision transformers. By inserting variational information bottlenecks on all attention-mediated writes to the residual stream -- without other architectural changes -- we train models with an explicit information cost and obtain a controllable spectrum from independent patch processing to fully expressive global attention. On ImageNet-100, we characterize how classification behavior and information routing evolve across this spectrum, and provide initial insights into how global visual representations emerge from local patch processing by analyzing the first attention heads that transmit information. By biasing learning toward solutions with constrained internal communication, our approach yields models that are more tractable for mechanistic analysis and more amenable to control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。