用指令引导推理,提升灾难后图像问答的准确性
Instruct-ICL: Instruction-Guided In-Context Learning for Post-Disaster Damage Assessment
- 让一个大模型生成任务指令,指导另一个模型进行分步推理
- 在FloodNet数据集上,准确率显著高于零样本基线
- 适合需要快速、可靠灾情分析的应急响应场景
自然灾害发生后,快速准确的情境感知对有效应对至关重要,分析延迟会严重影响决策。为灾后评估训练专用模型通常耗时且计算成本高,在时间紧迫的情况下不切实际。因此,预训练多模态大语言模型(MLLMs)成为灾后视觉问答(VQA)的有力替代方案,该任务旨在联合图像与文本推理回答结构化问题。尽管这些模型具备强大的多模态推理能力,其输出对提示设计敏感,限制了在真实灾情评估中的可靠性。本文研究结构化推理策略能否提升MLLM在灾后VQA中的可靠性。具体地,探索一种提示范式:由一个MLLM生成任务指令,作为链式思维(CoT)指引,供第二个MLLM在不同程度的上下文学习(ICL)中使用,从而结合显式推理引导与上下文示例。我们在FloodNet数据集上评估,并与零样本基线对比。结果表明,引入指令驱动的CoT推理能持续提升答案准确率。
原文摘要 · Abstract (English)
Rapid and accurate situational awareness is essential for effective response during natural disasters, where delays in analysis can significantly hinder decision-making. Training task-specific models for post-disaster assessment is often time-consuming and computationally expensive, making such approaches impractical in time-critical scenarios. Consequently, pretrained multimodal large language models (MLLMs) have emerged as a promising alternative for post-disaster visual question answering (VQA), a task that aims to answer structured questions about visual scenes by jointly reasoning over images and text. While these models demonstrate strong multimodal reasoning capabilities, their responses can be sensitive to prompt formulation, which can limit their reliability in real-world disaster assessment scenarios. In this paper, we investigate whether structured reasoning strategies can improve the reliability of pretrained MLLMs for post-disaster VQA. Specifically, we explore multiple prompting paradigms in which one MLLM is used to generate task-specific instructions that serve as Chain-of-Thought (CoT) guidance for a second MLLM. These instructions are incorporated during answer generation with varying degrees of in-context learning (ICL), enabling the model to leverage both explicit reasoning guidance and contextual examples. We conduct our evaluation on the FloodNet dataset and compare these approaches against a zero-shot baseline. Our results demonstrate that integrating instruction-driven CoT reasoning consistently improves answer accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。