用多模态模型隐层特征检测假内容，效果更好且通用性强。
Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics
- 利用预训练多模态模型的隐层特征区分真实与伪造内容。
- 线性分类器在音频和图像上达到顶尖性能，少样本下也有效。
- 方法高效快速，适合应对新出现的生成模型威胁。
生成模型在图像、文本等多个数据领域取得显著成果，但恶意用户常利用合成媒体传播虚假信息与深度伪造内容。因此，迫切需要具备鲁棒性和稳定性的假内容检测工具，尤其在新型生成模型层出不穷的背景下。现有方法大多训练判别真实与伪造信息的分类器，但通常仅在同类型生成器和数据模态间有效，对其他生成类别和数据域泛化能力差。为实现通用检测器，我们提出使用大规模预训练多模态模型进行生成内容检测。研究发现，这些模型的隐层编码天然蕴含真实与伪造之间的判别信息。基于此，我们证明：以这些特征训练的线性分类器，在多种模态下均能实现最先进的性能，同时计算效率高、训练速度快，并在少样本设置下依然有效。本工作主要聚焦音频与图像中的假内容检测，其表现超越或媲美强基线方法。
原文摘要 · Abstract (English)
Generative models achieve remarkable results in multiple data domains, including images and texts, among other examples. Unfortunately, malicious users exploit synthetic media for spreading misinformation and disseminating deepfakes. Consequently, the need for robust and stable fake detectors is pressing, especially when new generative models appear everyday. While the majority of existing work train classifiers that discriminate between real and fake information, such tools typically generalize only within the same family of generators and data modalities, yielding poor results on other generative classes and data domains. Towards a universal classifier, we propose the use of large pre-trained multi-modal models for the detection of generative content. Effectively, we show that the latent code of these models naturally captures information discriminating real from fake. Building on this observation, we demonstrate that linear classifiers trained on these features can achieve state-of-the-art results across various modalities, while remaining computationally efficient, fast to train, and effective even in few-shot settings. Our work primarily focuses on fake detection in audio and images, achieving performance that surpasses or matches that of strong baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。