通过动态加权融合图文特征，提升仇恨梗图检测准确率。
GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection
- 引入可学习投影头与动态门控融合机制，优化图文对齐。
- 在仇恨梗图数据集上达到0.66的AUROC，优于基线0.49。
- 仅35万参数，高效且适合实际部署场景。
检测多模态梗图中的仇恨内容面临独特挑战，因有害信息常源于看似无害的图像与文本之间的复杂交互。我们提出GatedCLIP，一种增强CLIP多模态能力的视觉-语言模型，通过专用架构改进以提升仇恨梗图检测性能。方法包括：将CLIP嵌入映射至任务优化语义空间的可学习投影头；自适应加权视觉与文本特征的动态门控融合机制；以及保持跨模态语义对齐的对比学习目标。在Hateful Memes数据集上的实验表明，GatedCLIP实现0.66的AUROC，显著优于原始CLIP基线（0.49），同时仅需35万可训练参数，计算效率高。
原文摘要 · Abstract (English)
Detecting hateful content in multimodal memes presents unique challenges, as harmful messages often emerge from the complex interplay between benign images and text. We propose GatedCLIP, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection. Our approach introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment. Experiments on the Hateful Memes dataset demonstrate that GatedCLIP achieves an AUROC of 0.66, substantially outperforming the CLIP baseline (AUROC 0.49) while maintaining computational efficiency with only 350K trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。