不依赖数据，直接分析CLIP权重，解释其视觉注意力机制
From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition
- 通过奇异向量分解直接分析模型权重，无需训练和数据
- 揭示每个注意力头的语义构成，实现精准概念编辑
- 适合研究模型内部机制或做无训练修改的开发者
随着视觉语言模型大规模部署，理解其内部机制变得愈发重要。现有可解释性方法主要依赖激活值，受数据集影响，易受数据偏见干扰，且通常仅提供粗粒度头部级解释。我们提出SITH（视觉变换器头部语义检查），一种完全无需数据、无需训练的框架，直接在权重空间分析CLIP的视觉变换器。针对每个注意力头，我们将值输出矩阵分解为奇异向量，并通过新算法COMP（一致正交匹配追踪）将其解释为稀疏、语义连贯的人类可理解概念组合。实验表明，SITH能生成一致且忠实的层内解释，通过重建保真度与可解释性验证。这使得我们可在不重新训练的前提下，通过权重空间编辑精确放大或抑制特定概念，提升下游性能。此外，我们用SITH研究模型适配，发现微调主要在重分配一个稳定的语义基底，而非学习全新特征。
原文摘要 · Abstract (English)
As vision-language models are deployed at scale, understanding their internal mechanisms becomes increasingly critical. Existing interpretability methods predominantly rely on activations, making them dataset-dependent, vulnerable to data bias, and often restricted to coarse head-level explanations. We introduce SITH (Semantic Inspection of Transformer Heads), a fully data-free, training-free framework that directly analyzes CLIP's vision transformer in weight space. For each attention head, we decompose its value-output matrix into singular vectors and interpret each one via COMP (Coherent Orthogonal Matching Pursuit), a new algorithm that explains them as sparse, semantically coherent combinations of human-interpretable concepts. We show that SITH yields coherent, faithful intra-head explanations, validated through reconstruction fidelity and interpretability experiments. This allows us to use SITH for precise, interpretable weight-space model edits that amplify or suppress specific concepts, improving downstream performance without retraining. Furthermore, we use SITH to study model adaptation, showing how fine-tuning primarily reweights a stable semantic basis rather than learning entirely new features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。