CLIP能从图像中提取出相机参数等细微痕迹，影响语义识别结果。
Processing and acquisition traces in visual encoders: What does CLIP know about your camera?
- 分析图像采集与处理过程中的细微特征在视觉模型中的编码方式。
- 发现这些隐藏痕迹可被准确还原，并显著影响分类性能。
- 适用于关注模型隐私泄露或数据溯源的研究者。
以往研究主要关注视觉编码器对图像变换和损坏的鲁棒性，尤其在测试时出现训练中未见的分布偏移导致性能下降的情况。本文换一个角度，研究那些人眼难以察觉、但源于图像采集或处理过程的细微参数。我们发现，这些参数在预训练视觉表征中被系统性地编码，且可被轻易恢复。更令人惊讶的是，这些痕迹的存在会对语义预测产生深远影响：当其与语义标签强相关或负相关时，会显著提升或降低模型表现。代码与数据已公开于 https://github.com/ryan-caesar-ramos/visual-encoder-traces。
原文摘要 · Abstract (English)
Prior work has analyzed the robustness of visual encoders to image transformations and corruptions, particularly in cases where such alterations are not seen during training. When this occurs, they introduce a form of distribution shift at test time, often leading to performance degradation. The primary focus has been on severe corruptions that, when applied aggressively, distort useful signals necessary for accurate semantic predictions. We take a different perspective by analyzing parameters of the image acquisition process and transformations that may be subtle or even imperceptible to the human eye. We find that such parameters are systematically encoded in the learned visual representations and can be easily recovered. More strikingly, their presence can have a profound impact, either positively or negatively, on semantic predictions. This effect depends on whether there is a strong correlation or anti-correlation between semantic labels and these acquisition-based or processing-based labels. Our code and data are available at: https://github.com/ryan-caesar-ramos/visual-encoder-traces
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。