用多跳思维链识别隐晦性别歧视梗图,提升模型理解力。
M3Hop-CoT: Misogynous Meme Identification with Multimodal Multi-hop Chain-of-Thought
- 构建多模态多跳思维链框架,融合图文语义与关系推理
- 在MAMI数据集上达最高宏平均F1分数,显著优于基线
- 适合研究网络暴力、AI伦理及多模态内容安全的学者
近年来,社交媒体上针对女性的仇恨言论,特别是通过性别歧视性梗图传播的现象日益严重。这类梗图常使用微妙且隐蔽的线索,使自动化检测极具挑战。尽管大语言模型(LLMs)通过思维链(CoT)提示在多模态任务中展现潜力,但往往忽视文化差异以及视觉模态中隐藏的情感和上下文知识。为此,我们提出一种多模态多跳思维链(M3Hop-CoT)框架,结合基于CLIP的分类器与融合实体-对象-关系的多模态思维链模块。该框架采用三步多模态提示策略,引导模型生成情感认知、目标意识和上下文知识。在SemEval-2022 Task 5(MAMI任务)数据集上的实证评估显示,该框架在宏平均F1分数上表现优异。此外,我们在多个基准梗图数据集上测试了模型泛化能力,全面验证了方法的有效性。
原文摘要 · Abstract (English)
In recent years, there has been a significant rise in the phenomenon of hate against women on social media platforms, particularly through the use of misogynous memes. These memes often target women with subtle and obscure cues, making their detection a challenging task for automated systems. Recently, Large Language Models (LLMs) have shown promising results in reasoning using Chain-of-Thought (CoT) prompting to generate the intermediate reasoning chains as the rationale to facilitate multimodal tasks, but often neglect cultural diversity and key aspects like emotion and contextual knowledge hidden in the visual modalities. To address this gap, we introduce a Multimodal Multi-hop CoT (M3Hop-CoT) framework for Misogynous meme identification, combining a CLIP-based classifier and a multimodal CoT module with entity-object-relationship integration. M3Hop-CoT employs a three-step multimodal prompting principle to induce emotions, target awareness, and contextual knowledge for meme analysis. Our empirical evaluation, including both qualitative and quantitative analysis, validates the efficacy of the M3Hop-CoT framework on the SemEval-2022 Task 5 (MAMI task) dataset, highlighting its strong performance in the macro-F1 score. Furthermore, we evaluate the model's generalizability by evaluating it on various benchmark meme datasets, offering a thorough insight into the effectiveness of our approach across different datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。