视觉模型会偷懒，通过照片元数据识别拍摄设备
Invisible Shortcuts: Why Vision Encoders Know Your Camera
- 发现模型依赖图像像素级隐藏的元数据线索
- 元数据相关性越强，分布偏移时性能下降越明显
- 可减少对未知元数据的敏感性，提升泛化能力
深度视觉模型会利用捷径，依赖与标注信号相关的线索。以往研究关注可见偏差，如物体-背景或纹理相关性。本文揭示了另一类捷径来源：嵌入在像素层面的不可见元数据痕迹，如图像处理和拍摄信息。我们假设，大规模语义监督（如ImageNet分类或LAION十亿级描述）在预训练过程中自然诱导出元数据与语义的相关性，使模型将低层信号转化为预测特征。通过引入可控的元数据-语义相关性，我们发现相关性越强，模型对元数据痕迹的敏感度越高，且在元数据分布偏移下性能下降越大。进一步探索预训练中及之后的缓解策略，可降低对目标元数据及未见元数据的敏感性，同时保持下游任务性能。元数据敏感性也有积极一面：部分解释了某些编码器在生成图像检测中的强能力；其缓解还能提升分布外泛化能力。
原文摘要 · Abstract (English)
Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。