用多模态分析识别隐晦仇恨梗图，效果优于传统方法。
Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
- 结合文字识别、视觉描述与问答，挖掘图文深层语义
- 在Facebook仇恨梗图数据集上准确率与AUC均超现有模型
- 适合研究网络暴力检测或内容安全的开发者使用
梗图广泛用于幽默和文化评论，但正被滥用于传播仇恨内容。由于其多模态特性，仇恨梗图常规避仅依赖文本或图像的检测系统，尤其当使用微妙或隐喻性表达时。为此，我们提出一种多模态仇恨检测框架，整合关键组件：OCR提取嵌入文本，图像描述生成中性视觉内容表征，子标签分类实现仇恨内容细粒度标注，RAG实现上下文相关知识检索，以及视觉问答（VQA）进行符号与语境线索的迭代分析。该框架能发现简单流水线难以捕捉的潜在信号。在Facebook仇恨梗图数据集上的实验表明，所提框架在准确率和AUC-ROC指标上均优于单模态及传统多模态模型。
原文摘要 · Abstract (English)
Memes are widely used for humor and cultural commentary, but they are increasingly exploited to spread hateful content. Due to their multimodal nature, hateful memes often evade traditional text-only or image-only detection systems, particularly when they employ subtle or coded references. To address these challenges, we propose a multimodal hate detection framework that integrates key components: OCR to extract embedded text, captioning to describe visual content neutrally, sub-label classification for granular categorization of hateful content, RAG for contextually relevant retrieval, and VQA for iterative analysis of symbolic and contextual cues. This enables the framework to uncover latent signals that simpler pipelines fail to detect. Experimental results on the Facebook Hateful Memes dataset reveal that the proposed framework exceeds the performance of unimodal and conventional multimodal models in both accuracy and AUC-ROC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。