融合社交上下文的多模态学习框架,提升灾情信息分类准确率。
A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method
- 构建图神经网络捕捉文本与图像的复杂关系及用户社交信息。
- 在两个数据集上分别提升9.45%和5.01%的F1分数。
- 适合应急响应、社交媒体分析与多模态学习研究者使用。
危机时刻,及时准确地分类社交媒体上的灾情信息对有效应对灾害和保障公众安全至关重要。人们常通过社交平台分享包含文本与视觉内容的多模态信息,但海量未经筛选的多样化数据使救援组织难以高效利用。现有方法往往忽视用户可信度、情感语境及社交互动信息,影响分类精度。为此,我们提出CrisisSpot,采用基于图神经网络的方法,建模文本与视觉模态间的关系,并融合用户中心与内容中心的社交上下文特征。引入逆向双重嵌入注意力(IDEA),捕捉数据中和谐与对比模式,增强多模态交互。同时发布土耳其-叙利亚地震数据集TSEqD,涵盖10,352个标注样本。实验表明,CrisisSpot在公开的CrisisMMD数据集和TSEqD数据集上分别相较当前最优方法提升平均F1分数9.45%和5.01%。
原文摘要 · Abstract (English)
In times of crisis, the prompt and precise classification of disaster-related information shared on social media platforms is crucial for effective disaster response and public safety. During such critical events, individuals use social media to communicate, sharing multimodal textual and visual content. However, due to the significant influx of unfiltered and diverse data, humanitarian organizations face challenges in leveraging this information efficiently. Existing methods for classifying disaster-related content often fail to model users' credibility, emotional context, and social interaction information, which are essential for accurate classification. To address this gap, we propose CrisisSpot, a method that utilizes a Graph-based Neural Network to capture complex relationships between textual and visual modalities, as well as Social Context Features to incorporate user-centric and content-centric information. We also introduce Inverted Dual Embedded Attention (IDEA), which captures both harmonious and contrasting patterns within the data to enhance multimodal interactions and provide richer insights. Additionally, we present TSEqD (Turkey-Syria Earthquake Dataset), a large annotated dataset for a single disaster event, containing 10,352 samples. Through extensive experiments, CrisisSpot demonstrated significant improvements, achieving an average F1-score gain of 9.45% and 5.01% compared to state-of-the-art methods on the publicly available CrisisMMD dataset and the TSEqD dataset, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。