arXiv:2507.05165cs.CV2025-07

用差异注意力提升多模态危机数据分类,让救援更准更快。

Differential Attention for Multimodal Crisis Event Analysis

  • 引入差异注意力机制,动态聚焦关键信息,过滤无关内容。
  • 在CrisisMMD数据集上,分类准确率超越现有模型。
  • 适合灾难响应、人道主义救援等需要实时分析的场景。

社交媒体在危机事件中可提供宝贵信息,用户发布的多模态数据对实时人道主义响应至关重要。然而,从海量嘈杂数据中有效提取有意义信息并融合异构数据仍具挑战。本文探索视觉语言模型(VLMs)与先进融合策略,提升三种任务下的危机数据分类性能。利用LLaVA生成文本增强图文对齐,并采用无需任务微调的CLIP视觉与文本嵌入,表现优于传统模型。为进一步优化多模态融合,引入引导交叉注意力(Guided CA),结合差异注意力机制,通过强调关键特征、抑制冗余内容,实现更好的特征对齐。在CrisisMMD基准数据集上的实验表明,预训练VLM、丰富文本描述与自适应融合策略的组合,在三类任务中持续优于当前最优模型,显著提升分类准确率与模型可解释性,助力更可靠的灾情响应系统。代码已开源。

原文摘要 · Abstract (English)

Social networks can be a valuable source of information during crisis events. In particular, users can post a stream of multimodal data that can be critical for real-time humanitarian response. However, effectively extracting meaningful information from this large and noisy data stream and effectively integrating heterogeneous data remains a formidable challenge. In this work, we explore vision language models (VLMs) and advanced fusion strategies to enhance the classification of crisis data in three different tasks. We incorporate LLaVA-generated text to improve text-image alignment. Additionally, we leverage Contrastive Language-Image Pretraining (CLIP)-based vision and text embeddings, which, without task-specific fine-tuning, outperform traditional models. To further refine multimodal fusion, we employ Guided Cross Attention (Guided CA) and combine it with the Differential Attention mechanism to enhance feature alignment by emphasizing critical information while filtering out irrelevant content. Our results show that while Differential Attention improves classification performance, Guided CA remains highly effective in aligning multimodal features. Extensive experiments on the CrisisMMD benchmark data set demonstrate that the combination of pretrained VLMs, enriched textual descriptions, and adaptive fusion strategies consistently outperforms state-of-the-art models in classification accuracy, contributing to more reliable and interpretable models for three different tasks that are crucial for disaster response. Our code is available at https://github.com/Munia03/Multimodal_Crisis_Event.

多模态危机分析注意力机制视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。