用图像重建解析视觉模型内部特征，发现训练方式影响信息保留
Image Reconstruction as a Tool for Feature Analysis
- 通过重建图像反推特征表示，比较不同训练目标的效果
- 图像任务预训练的编码器保留更多原始图像信息
- 特征空间旋转可预测控制颜色变化，适合研究模型内部机制
视觉编码器在现代应用中广泛使用，从纯视觉模型到视觉-语言等多模态系统。尽管表现优异，其内部特征表示仍不清晰。本文提出一种基于图像重建的视觉特征分析新方法，对比了仅训练目标不同的两个模型家族SigLIP和SigLIP2，发现基于图像任务预训练的编码器比非图像任务（如对比学习）训练的编码器保留显著更多的图像信息。进一步将该方法应用于多种视觉编码器，按特征表示的信息量进行排序。最后证明，操纵特征空间可导致重建图像产生可预测的变化，揭示正交旋转（而非空间变换）控制颜色编码。该方法适用于任意视觉编码器，有助于理解其特征空间内部结构。代码与模型权重已在GitHub公开。
原文摘要 · Abstract (English)
Vision encoders are increasingly used in modern applications, from vision-only models to multimodal systems such as vision-language models. Despite their remarkable success, it remains unclear how these architectures represent features internally. Here, we propose a novel approach for interpreting vision features via image reconstruction. We compare two related model families, SigLIP and SigLIP2, which differ only in their training objective, and show that encoders pre-trained on image-based tasks retain significantly more image information than those trained on non-image tasks such as contrastive learning. We further apply our method to a range of vision encoders, ranking them by the informativeness of their feature representations. Finally, we demonstrate that manipulating the feature space yields predictable changes in reconstructed images, revealing that orthogonal rotations (rather than spatial transformations) control color encoding. Our approach can be applied to any vision encoder, shedding light on the inner structure of its feature space. The code and model weights to reproduce the experiments are available in GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。