对比视频与图片的仇恨内容检测，发现现有融合方法效果差异大。
Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content
- 用简单嵌入融合法在视频上表现最佳，提升9.9%准确率。
- 面对图文混合的梗图，现有方法因无法捕捉复杂关系而失效。
- 研究提示需针对不同模态设计专用检测架构，适合安全与平台方参考。
社交媒体平台使仇恨内容在文本、音频和视觉等多模态间传播，亟需有效的检测方法。尽管近期方法在单一模态上表现良好，但其在多模态组合下的效果仍不明确。本文系统分析了基于融合的多模态仇恨检测方法在视频与图像内容上的表现。全面评估显示显著的模态特异性限制：简单嵌入融合在视频数据集HateMM上达到当前最优性能，F1分数提升9.9个百分点；但在涉及复杂图文关系的梗图数据集Hateful Memes上表现不佳。通过详尽的消融实验与错误分析，我们揭示现有融合方法难以捕捉细微的跨模态交互，尤其在存在良性混淆因素时。研究结果为构建更鲁棒的仇恨检测系统提供了关键洞见，并强调需考虑模态特异性架构设计。代码已公开于https://github.com/gak97/Video-vs-Meme-Hate。
原文摘要 · Abstract (English)
Social media platforms enable the propagation of hateful content across different modalities such as textual, auditory, and visual, necessitating effective detection methods. While recent approaches have shown promise in handling individual modalities, their effectiveness across different modality combinations remains unexplored. This paper presents a systematic analysis of fusion-based approaches for multimodal hate detection, focusing on their performance across video and image-based content. Our comprehensive evaluation reveals significant modality-specific limitations: while simple embedding fusion achieves state-of-the-art performance on video content (HateMM dataset) with a 9.9% points F1-score improvement, it struggles with complex image-text relationships in memes (Hateful Memes dataset). Through detailed ablation studies and error analysis, we demonstrate how current fusion approaches fail to capture nuanced cross-modal interactions, particularly in cases involving benign confounders. Our findings provide crucial insights for developing more robust hate detection systems and highlight the need for modality-specific architectural considerations. The code is available at https://github.com/gak97/Video-vs-Meme-Hate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。