无需像素标注,用文本提示定位图像篡改边缘。
SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization
- 设计边界感知的文本提示,让CLIP关注篡改边缘而非物体语义。
- 在多个公开数据集上实现当前最优定位性能,准确率显著提升。
- 适合需要低成本弱监督图像篡改检测的研究者与应用开发者。
恶意图像篡改威胁公共安全,亟需高效定位方法。现有方法依赖昂贵的像素级标注,而弱监督方法仅使用图像级二值标签,聚焦全局分类,常忽略对精确定位至关重要的局部边缘特征。我们观察到篡改边界的特征变化显著大于内部区域。为此,提出在CLIP中采用语义无关提示学习(SAPL),通过学习不包含语义信息的、以边界为中心的文本提示,使CLIP多模态相似性聚焦于篡改边缘而非高层语义。SAPL结合两个互补模块:边缘感知上下文提示学习(ECPL)利用增强边缘的图像特征,通过注意力机制生成可学习的文本提示,将非语义信息嵌入文本特征,引导CLIP关注篡改边缘;分层边缘对比学习(HECL)提取真实与篡改边缘块,利用对比学习增强二者区分能力。最终基于处理后的相似度图预测篡改区域。在多个公开基准上的实验证明,SAPL显著优于现有方法,达到当前最优定位性能。
原文摘要 · Abstract (English)
Malicious image manipulation threatens public safety and requires efficient localization methods. Existing approaches depend on costly pixel-level annotations which make training expensive. Existing weakly supervised methods rely only on image-level binary labels and focus on global classification, often overlooking local edge cues that are critical for precise localization. We observe that feature variations at manipulated boundaries are substantially larger than in interior regions. To address this gap, we propose Semantic-Agnostic Prompt Learning (SAPL) in CLIP, which learns text prompts that intentionally encode non-semantic, boundary-centric cues so that CLIPs multimodal similarity highlights manipulation edges rather than high-level object semantics. SAPL combines two complementary modules Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL) to exploit edge information in both textual and visual spaces. The proposed ECPL leverages edge-enhanced image features to generate learnable textual prompts via an attention mechanism, embedding semantic-irrelevant information into text features, to guide CLIP focusing on manipulation edges. The proposed HECL extract genuine and manipulated edge patches, and utilize contrastive learning to boost the discrimination between genuine edge patches and manipulated edge patches. Finally, we predict the manipulated regions from the similarity map after processing. Extensive experiments on multiple public benchmarks demonstrate that SAPL significantly outperforms existing approaches, achieving state-of-the-art localization performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。