用小波分析ViT如何组合视觉特征，发现低层特征可近似拼接。
Exploring Compositionality in Vision Transformers using Wavelet Representations
- 用离散小波变换提取图像局部基元，模拟特征组合机制。
- 一层小波分解的基元在编码器中能近似重构原图表示。
- 为理解ViT信息组织方式提供新视角，适合关注模型可解释性者。
尽管对Transformer模型的理解主要基于语言任务，本文从组合性角度研究视觉Transformer（ViT）编码器所学表征。提出一种框架，类比此前在表征学习中测量组合性的方法，测试ViT编码器中的组合性。关键在于使用离散小波变换（DWT），这是一种简单而有效的视觉输入依赖基元获取工具。通过检验组合表征再现原始图像表征的能力，实证评估组合性在表征空间中的成立程度。结果表明，单层DWT分解得到的基元能在隐空间近似组合，形成编码器表征，为理解ViT的信息结构提供了新视角。
原文摘要 · Abstract (English)
While insights into the workings of the transformer model have largely emerged by analysing their behaviour on language tasks, this work investigates the representations learnt by the Vision Transformer (ViT) encoder through the lens of compositionality. We introduce a framework, analogous to prior work on measuring compositionality in representation learning, to test for compositionality in the ViT encoder. Crucial to drawing this analogy is the Discrete Wavelet Transform (DWT), which is a simple yet effective tool for obtaining input-dependent primitives in the vision setting. By examining the ability of composed representations to reproduce original image representations, we empirically test the extent to which compositionality is respected in the representation space. Our findings show that primitives from a one-level DWT decomposition produce encoder representations that approximately compose in latent space, offering a new perspective on how ViTs structure information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。