通过图神经网络聚焦视频中仇恨内容片段,提升检测精度与可解释性。
Multimodal Hate Detection Using Dual-Stream Graph Neural Networks
- 将视频拆分为片段构建实例图,捕捉局部仇恨特征
- 用互补权重图突出仇恨片段,实现重点内容强调
- 结构化建模跨模态关系,适合需要透明决策的场景
仇恨视频对网络环境与现实社会构成严重威胁,亟需高效检测方法。尽管多模态融合方法优于单模态,但通常忽略哪怕微小的仇恨内容也决定视频类别这一关键事实,普遍对所有内容一视同仁,未能突出仇恨部分。同时,现有方法难以系统捕捉视频内部结构信息,限制了多模态融合效果。为此,我们提出一种新型双流图神经网络模型:首先将视频分割为多个片段,构建实例图以提取片段级特征;再通过互补权重图为各特征分配重要性权重,强化仇恨片段;最后融合权重与特征生成视频标签。该模型基于图结构框架,系统建模模态内与跨模态间的结构关系。在公开数据集上的大量实验表明,本模型在仇恨视频分类任务中达到当前最优性能,并具备强可解释性。代码已开源:https://github.com/Multimodal-Intelligence-Lab-MIL/MultiHateGNN。
原文摘要 · Abstract (English)
Hateful videos present serious risks to online safety and real-world well-being, necessitating effective detection methods. Although multimodal classification approaches integrating information from several modalities outperform unimodal ones, they typically neglect that even minimal hateful content defines a video's category. Specifically, they generally treat all content uniformly, instead of emphasizing the hateful components. Additionally, existing multimodal methods cannot systematically capture structured information in videos, limiting the effectiveness of multimodal fusion. To address these limitations, we propose a novel multimodal dual-stream graph neural network model. It constructs an instance graph by separating the given video into several instances to extract instance-level features. Then, a complementary weight graph assigns importance weights to these features, highlighting hateful instances. Importance weights and instance features are combined to generate video labels. Our model employs a graph-based framework to systematically model structured relationships within and across modalities. Extensive experiments on public datasets show that our model is state-of-the-art in hateful video classification and has strong explainability. Code is available: https://github.com/Multimodal-Intelligence-Lab-MIL/MultiHateGNN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。