分析9种增强对CLIP嵌入的影响，揭示哪些操作最易引发表征偏移。
Embedding Shift Dissection on CLIP: Effects of Augmentations on VLM's Representation Learning
- 系统测试9种图像增强对CLIP嵌入的扰动效果。
- 噪声、透视变换和缩放偏移导致嵌入变化最剧烈。
- 结果有助于提升视觉语言模型的可解释性与抗攻击能力。
理解在不同数据增强下，像CLIP这样的视觉语言模型（VLM）的表征偏移，有助于机制可解释性研究。本研究考察了9种常见增强技术对CLIP嵌入的影响：噪声、模糊、色彩抖动、缩放旋转、翻转、弹性与透视变换、随机亮度对比度调整，以及像素块粗略掩码。我们从注意力图、局部块、边缘、细节保留度、余弦相似度、L2距离、成对距离和树状聚类等角度分析嵌入偏移，并对样本图像进行定性分析。结果表明，噪声、透视变换和缩放偏移对嵌入影响最为显著。该研究为未来视觉语言模型的鲁棒性分析与机械可解释性研究提供了坚实基础。代码见：https://github.com/ashimdahal/clip-shift-analysis。
原文摘要 · Abstract (English)
Understanding the representation shift on Vision Language Models like CLIP under different augmentations provides valuable insights on Mechanistic Interpretability. In this study, we show the shift on CLIP's embeddings on 9 common augmentation techniques: noise, blur, color jitter, scale and rotate, flip, elastic and perspective transforms, random brightness and contrast, and coarse dropout of pixel blocks. We scrutinize the embedding shifts under similarity on attention map, patch, edge, detail preservation, cosine similarity, L2 distance, pairwise distance and dendrogram clusters and provide qualitative analysis on sample images. Our findings suggest certain augmentations like noise, perspective transform and shift scaling have higher degree of drastic impact on embedding shift. This study provides a concrete foundation for future work on VLM's robustness for mechanical interpretation and adversarial data defense. The code implementation for this study can be found on \href{https://github.com/ashimdahal/clip-shift-analysis}{https://github.com/ashimdahal/clip-shift-analysis}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。