将图像篡改检测从掩码升级到像素级语义理解,更精准识别真实编辑区域。
From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering
- 提出基于像素和语义的篡改分类新体系,涵盖替换、删除、拼接等操作类型。
- 构建首个带像素级标注与类别监督的新基准,支持精确定位与语义分类。
- 引入语言描述与意义感知评估,适合研究可解释性与细粒度篡改检测者。
现有篡改检测基准主要依赖物体掩码,严重偏离真实编辑信号:掩码内许多像素未被修改或仅轻微变动,而掩码外细微但关键的修改却被视为自然。本文将视觉语言模型(VLM)图像篡改检测从粗粒度区域标签转向像素级、语义与语言感知的任务。首先,提出涵盖替换、移除、拼接、修复、属性更改、着色等编辑原语及其语义类别的新分类体系,连接低层变化与高层理解。其次,发布首个包含像素级篡改图和配对类别监督的新基准,支持统一协议下的检测与分类评估。第三,设计训练框架与评估指标,通过定位精度量化像素级正确性,衡量预测置信度与真实编辑强度;并引入语义感知分类与自然语言描述,评估篡改意义理解能力。我们还重新评估现有强分割/定位基线在最新强检测器上的表现,揭示掩码仅指标存在严重过评与漏评,暴露对微小编辑和掩码外变化的失败模式。该框架推动领域从掩码走向像素、语义与语言描述,建立篡改定位、语义分类与描述的严格标准。代码与数据集见 https://github.com/VILA-Lab/PIXAR。
原文摘要 · Abstract (English)
Existing tampering detection benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural language descriptions for the predicted regions. We also re-evaluate the existing strong segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and benchmark data are available at https://github.com/VILA-Lab/PIXAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。