不靠梯度更新,用谱分析检测生成图像伪造痕迹。
Need We Teach Foundation Models What is a Generative Image? Gradient-Free Generative Artifact Detection via Analytic Spectral Adaptation

- 将检测转为无监督异常识别,利用冻结模型的稳定特征空间
- 在跨域零样本测试中,对文本生成图像的伪造检测准确率显著更高
- 支持边缘部署和在线更新,适合隐私敏感场景
通过梯度优化微调基础模型来检测生成伪影会破坏其内在表示。在小样本上优化时,模型易陷入局部领域捷径;在专用数据上大规模微调权重会引入错误归纳偏置,在高维特征空间引发可测量的 $/mathcal{L}_2$ 范数扰动,我们称之为锚点漂移。非线性激活放大此漂移,损害对未见领域的零样本伪造检测能力。本文提出一种无梯度方法,将检测从二分类重构为分布外(OOD)异常度量问题。以冻结的基础模型作为稳定坐标系,通过解析解耦注意力加权空间矩与感知不一致的正交投影,建立真实的视觉流形绝对自然锚点。在极端零样本设置下(训练于人脸伪造,测试于通用文生图),本方法显著优于基于梯度优化的范式。无需反向传播的前向计算与线性求解器实现硬件无关、低延迟边缘校准。此外,Sherman-Morrison 公式支持针对新攻击的瞬时在线学习,并可通过协方差增量传输实现隐私保护的联邦协作。
原文摘要 · Abstract (English)
Adapting foundation models to detect generative artifacts via gradient-based updates compromises their intrinsic representations. Under optimization on limited samples, models overfit to local domain shortcuts. Fine-tuning massive weights on specialized data introduces erroneous inductive biases, inducing a measurable $\mathcal{L}_2$ norm perturbation in the high-dimensional feature space -- a phenomenon we formalize as anchor drift. Amplified by nonlinear activations, this drift impairs zero-shot forgery detection across unseen domains.We propose a gradient-free methodology reframing detection from binary classification to an out-of-distribution (OOD) anomaly measurement problem. Treating a frozen foundation model as a stable coordinate system, we establish an absolute natural anchor on the real visual manifold by analytically decoupling statistical and semantic deviations, derived from attention-weighted spatial moments and orthogonal projection of perceptual inconsistencies. Evaluated in an extreme zero-shot setting (trained on face forgeries, tested on universal Text-to-Image generations), our method significantly outperforms gradient-optimized paradigms. Backpropagation-free forward passes and linear solvers enable hardware-agnostic, edge-deployable calibration with minimal latency. Furthermore, the Sherman-Morrison formula unlocks instantaneous online learning against novel attacks and enables privacy-preserving federated collaboration via covariance delta transmission.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。