融合多模态感知与检索增强的灾损评估系统,提升判断准确性与鲁棒性。
Integrated Multimodal AI System for Retrieval-Augmented Reasoning, Object Sensing, and Damage Analysis

- 用检索增强生成技术结合项目文档,减少大模型幻觉。
- 图谱式检索比向量检索更擅长跨文档推理,提升灾损分析质量。
- 红外+无线信号感知弥补可见光不足,适合复杂环境灾后检测。
本文提出一种统一的多模态AI系统,用于灾损评估,整合了检索增强生成(RAG)模型、热谱感知、视觉基础模型流水线及探索性无线信号传感。通过本地部署的语言模型结合项目专属文档进行检索增强,引入专业灾损等级分类标准,有效降低推理时的幻觉。对比静态少样本提示,动态检索显著提升内容准确性和事实一致性。进一步比较基于向量的RAG与基于实体-关系抽取构建的知识图谱变体,发现图谱检索在需跨文档推理的灾损问题上表现更优,推动采用密集、稀疏与图感知混合检索策略。针对可见光遥感在恶劣光照和天气下的局限性,采用红外(IR)/热成像实现目标检测与分割,生成候选检测结果,提升多种物体的分割精度。可见光与红外配对跟踪实验揭示了各自失效模式,促使采用多模态融合以增强目标检测与灾损分析的鲁棒性。利用视觉基础模型与视觉-语言模型生成合成灾损图像,并高精度分类灾损等级,支持下游评估模型的训练与验证。最后,探索性无线传感展现出在传统遥感失效场景下探测存在、运动及灾后环境变化的潜力。
原文摘要 · Abstract (English)
This work presents a unified multimodal AI system for damage assessment that integrates retrieval-augmented generation (RAG) models, thermal spectrum perception, vision foundation model pipelines, and exploratory wireless signal sensing. A RAG component is developed to ground a locally hosted language model in project-specific documentation, including specialized damage level classification criteria to mitigate hallucinations during inference. Controlled comparisons against static few-shot prompting demonstrate that dynamic retrieval improves grounding and factual consistency. We further compare vector-based RAG with a knowledge graph variant constructed via entity-relation extraction, and show that graph-based retrieval produces stronger responses for damage assessment queries requiring cross-document reasoning, motivating hybrid dense, sparse, and graph-aware retrieval. To address limitations of EO imagery under adverse lighting and weather conditions, infrared (IR)/thermal sensing is employed for object detection and segmentation. Our detectors generate candidate detections, yielding improved segmentation of a broad array of objects. Paired IR versus visible spectrum tracking experiments reveal failure modes, motivating multimodal fusion for robust object detection and damage analysis. Vision foundation and vision-language models are leveraged to generate synthetic damage imagery and classify damage severity with high accuracy, supporting training and validation of downstream damage assessment models. Finally, exploratory Wireless-based sensing demonstrates potential to detect presence, motion, and post-event environmental changes where EO and IR sensing are ineffective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。