通过通道与模态融合,提升仇恨视频的多模态检测效果
Enhanced Multimodal Hate Video Detection via Channel-wise and Modality-wise Fusion
- 设计通道与模态级融合机制,增强跨模态特征交互
- 在真实数据集上准确率、F1等指标显著优于5个基线模型
- 适合关注多模态内容安全与仇恨言论检测的研究者
TikTok和YouTube等平台视频内容的快速增长推动了信息传播,但也助长了仇恨视频的扩散。尽管已有大量努力应对仇恨言论,但因其常具隐含性,检测仍具挑战。现有方法多依赖单模态,难以捕捉不同模态间的互补特征;虽有部分多模态方法,却未能有效整合时序动态与模态间交互。本文提出CMFusion模型,采用通道与模态级融合机制,先用预训练模型提取文本、音频、视频特征,再引入时序交叉注意力捕获音视频流间依赖关系,最后通过融合模块生成高信息量表示。在真实数据集上的实验表明,CMFusion在准确率、精确率、召回率和F1分数上均显著超越五个主流基线。消融实验与参数分析进一步验证了设计有效性。源代码将公开于https://github.com/EvelynZ10/cmfusion。
原文摘要 · Abstract (English)
The rapid rise of video content on platforms such as TikTok and YouTube has transformed information dissemination, but it has also facilitated the spread of harmful content, particularly hate videos. Despite significant efforts to combat hate speech, detecting these videos remains challenging due to their often implicit nature. Current detection methods primarily rely on unimodal approaches, which inadequately capture the complementary features across different modalities. While multimodal techniques offer a broader perspective, many fail to effectively integrate temporal dynamics and modality-wise interactions essential for identifying nuanced hate content. In this paper, we present CMFusion, an enhanced multimodal hate video detection model utilizing a novel Channel-wise and Modality-wise Fusion Mechanism. CMFusion first extracts features from text, audio, and video modalities using pre-trained models and then incorporates a temporal cross-attention mechanism to capture dependencies between video and audio streams. The learned features are then processed by channel-wise and modality-wise fusion modules to obtain informative representations of videos. Our extensive experiments on a real-world dataset demonstrate that CMFusion significantly outperforms five widely used baselines in terms of accuracy, precision, recall, and F1 score. Comprehensive ablation studies and parameter analyses further validate our design choices, highlighting the model's effectiveness in detecting hate videos. The source codes will be made publicly available at https://github.com/EvelynZ10/cmfusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。