构建首个多模态仇恨与宣传表情包解释数据集，提升检测与解释能力。
MemeIntel: Explainable Detection of Propagandistic and Hateful Memes
- 提出分阶段优化策略，联合训练视觉语言模型增强解释性。
- 在阿拉伯语和英语数据集上准确率分别提升1.4%和2.2%。
- 适合关注虚假信息、仇恨言论检测与可解释AI的研究者。
社交媒体上多模态内容的泛滥给理解与治理虚假信息、仇恨言论及宣传等复杂、依赖上下文的问题带来了巨大挑战。尽管已有资源和方法被用于自动检测，但对标签识别与解释性推理的联合建模关注有限，常导致联合训练时性能下降。为此，我们构建了MemeXplain——首个针对阿拉伯语宣传性表情包和英语仇恨性表情包的大规模解释增强数据集。为解决该任务，我们提出一种多阶段优化方法并训练视觉-语言模型（VLMs）。结果表明，该策略显著提升了标签检测与解释生成质量，在ArMeme和Hateful Memes数据集上分别实现约1.4%（准确率）和2.2%（准确率）的绝对提升，优于当前最优方法。为促进复现与后续研究，我们将公开MemeXplain数据集与代码（https://github.com/MohamedBayan/MemeIntel）。
原文摘要 · Abstract (English)
The proliferation of multimodal content on social media presents significant challenges in understanding and moderating complex, context-dependent issues such as misinformation, hate speech, and propaganda. While efforts have been made to develop resources and propose new methods for automatic detection, limited attention has been given to jointly modeling label detection and the generation of explanation-based rationales, which often leads to degraded classification performance when trained simultaneously. To address this challenge, we introduce MemeXplain, an explanation-enhanced dataset for propagandistic memes in Arabic and hateful memes in English, making it the first large-scale resource for these tasks. To solve these tasks, we propose a multi-stage optimization approach and train Vision-Language Models (VLMs). Our results show that this strategy significantly improves both label detection and explanation generation quality over the base model, outperforming the current state-of-the-art with an absolute improvement of ~1.4% (Acc) on ArMeme and ~2.2% (Acc) on Hateful Memes. For reproducibility and future research, we aim to make the MemeXplain dataset and scripts publicly available (https://github.com/MohamedBayan/MemeIntel).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。