用分割引导评分提升图像全局不一致检测能力
SGS: Segmentation-Guided Scoring for Global Scene Inconsistencies
- 通过分割掩码分离前景背景，实现区域感知的语义一致性评分
- 在DGM4数据集上使全局篡改检测准确率显著提升，尤其改善上下文错位场景
- 无需重训练、计算开销极低，适合实际部署于多模态内容审核
我们扩展了当前最先进的多模态篡改检测模型HAMMER，以应对前景-背景（FG-BG）不一致等全局场景问题。尽管HAMMER在DGM4数据集上表现优异，但在主体与背景语义不符时持续失效。诊断发现该缺陷源于标签空间偏差、局部注意力聚焦及虚假文本-前景对齐。为解决此问题且无需重训练，我们提出轻量级的分割引导评分（SGS）流程：利用人像/人脸分割掩码划分前景与背景区域，通过联合视觉-语言模型提取嵌入，并计算区域感知的一致性得分。这些得分与HAMMER原始预测融合，显著提升二分类检测、定位精度和词级别解释能力。SGS仅需推理阶段操作，计算开销可忽略，大幅增强对全局篡改的鲁棒性。本工作凸显区域感知推理在多模态虚假信息检测中的关键作用。相关分割与评分脚本已开源：https://github.com/Gaganx0/HAMMER-sgs。
原文摘要 · Abstract (English)
We extend HAMMER, a state-of-the-art model for multimodal manipulation detection, to handle global scene inconsistencies such as foreground-background (FG-BG) mismatch. While HAMMER achieves strong performance on the DGM4 dataset, it consistently fails when the main subject is contextually misplaced into an implausible background. We diagnose this limitation as a combination of label-space bias, local attention focus, and spurious text-foreground alignment. To remedy this without retraining, we propose a lightweight segmentation-guided scoring (SGS) pipeline. SGS uses person/face segmentation masks to separate foreground and background regions, extracts embeddings with a joint vision-language model, and computes region-aware coherence scores. These scores are fused with HAMMER's original prediction to improve binary detection, grounding, and token-level explanations. SGS is inference-only, incurs negligible computational overhead, and significantly enhances robustness to global manipulations. This work demonstrates the importance of region-aware reasoning in multimodal disinformation detection. We release scripts for segmentation and scoring at https://github.com/Gaganx0/HAMMER-sgs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。