让大模型同时懂语义又识造假,还能自适应判断
Deep Residual Injection for Full-Spectrum Forensic Signal Perception in Multimodal Large Language Models

- 在中间层注入伪造信号残差,不破坏原有语义理解
- 在多个数据集上达到当前最优检测性能
- 适合需要兼顾语义与图像真伪识别的场景
多模态大模型(MLLM)在司法鉴定中因强大的语义理解能力被广泛应用。随着生成图像日益逼真,仅靠语义不一致已难以可靠检测。这引发一个关键问题:MLLM能否实现全谱段伪造信号感知,即在不损失预训练语义知识的前提下捕捉低层级生成痕迹?我们对MLLM的层次化分析表明,语义信息主要在早期到中期层形成,而直接微调会破坏这些表示。基于此,我们提出深度视觉残差MLLM(Deep-VRM),通过在中间层引入特定于伪造的视觉信号残差路径,将其与语义标记表示融合并传递至后续可训练层。这使得后期层能联合建模语义推理与信号级伪造线索,且模型能根据输入自适应地利用不同层次的伪造信号,实现鲁棒且泛化性强的检测效果。大量实验表明,该方法在多数基准上达到领先水平。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have been increasingly adopted in forensics for their robust semantic understanding. As AI-generated images become realistic, semantic-level inconsistencies alone are often insufficient for reliable detection. This motivates a critical question: whether MLLMs can achieve full-spectrum forensic signal perception, i.e., capturing low-level generator artifacts without sacrificing pre-trained semantic knowledge. We further perform a layer-wise analysis of forensic signal perception in MLLMs, showing that semantic information is primarily formed in the early-to-middle layers, whereas direct fine-tuning for artifact learning disrupts these semantic representations. Based on this insight, we propose Deep Visual Residual MLLM (Deep-VRM) to preserve early semantic processing while injecting artifact-specific visual signals as a residual path into an intermediate layer, where they are fused with semantic token representations and propagated through subsequent trainable layers. This enables later layers to jointly model semantic reasoning and signal-level forensic cues, and surprisingly, the model learns to adaptively leverage different levels of forensic signals depending on the input, achieving robust and generalizable detection performance. Extensive experiments show that our method achieves state-of-the-art across most benchmarks. The code and data are available at https://github.com/KQL11/Deep-VRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。