arXiv:2608.07562cs.CVcs.LG2026-08中稿 · ICML

用可解释性指导微调,让视觉语言模型实现厘米级洪水深度估计。

Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

  • 通过机制可解释性分析,仅微调关键注意力层提升效率。
  • 模型达0.40厘米MAE,真实场景准确率98.62%。
  • 适合需要高效高精度洪水监测的智慧城市项目。

城市内涝对交通基础设施构成日益严重的威胁,但目前尚无实时、街景级厘米级洪水深度估计系统。本文提出三种针对街景图像连续洪水深度估计的视觉语言模型:FloodLlama-Dense(全量微调QLoRA基线)、FloodLlama-MI5和FloodLlama-MI6(基于可解释性分析的稀疏微调变体,分别仅微调前五和六层因果相关交叉注意力层)。训练使用约61万张来自281万张合成图像数据集的子集,该数据集在Unreal Engine 5中生成,包含单车与混合车辆子集,深度增量分别为5厘米和1厘米,涵盖七种车型、四种天气条件,洪水深度0至40厘米。FloodLlama-Dense实现MAE 0.40 cm,RMSE 1.97 cm,Acc@5cm 97.59%。机制可解释性分析结合线性探测、逻辑透镜、中心核对齐(CKA)与交叉注意力熵,揭示两阶段适配模式:第13-22层重构视觉表征,深度在第23层首次线性可解码。FloodLlama-MI5与MI6仅微调其中五或六层,参数量减少86-88%(655万–786万对比5440万),精度损失极小。在真实世界基准测试中,FloodLlama-MI6准确率达98.62%,优于已发表的STURM-FloodDepth基线(86.61%)。

原文摘要 · Abstract (English)

Urban flooding poses an escalating threat to transportation infrastructure, yet no operational system provides real-time, street-level flood-depth estimates at centimeter resolution. This paper presents three vision-language models fine-tuned for continuous flood-depth estimation from street-level imagery: FloodLlama-Dense, a fully fine-tuned QLoRA baseline, and FloodLlama-MI5 and FloodLlama-MI6, interpretability-guided sparse variants that fine-tune only the top five and six causally relevant cross-attention layers identified through mechanistic interpretability analysis, respectively. Training uses an approximately 610,000-image subset of a 2.81-million-image synthetic corpus generated in Unreal Engine 5. The dataset combines single-vehicle subsets with 5 cm depth increments and mixed-vehicle subsets with 1 cm depth increments, spanning seven vehicle types, four weather conditions, and flood depths from 0 to 40 cm. FloodLlama-Dense achieves an MAE of 0.40 cm, an RMSE of 1.97 cm, and an Acc@5cm of 97.59%. Mechanistic interpretability analysis combining linear probing, logit lens, centered kernel alignment (CKA), and cross-attention entropy reveals a two-stage adaptation pattern: layers L13-L22 restructure visual representations, while depth first becomes linearly decodable at layer L23. FloodLlama-MI5 and FloodLlama-MI6 leverage this insight by fine-tuning only five or six of the eight cross-attention layers, achieving an 86-88% reduction in trainable parameters (6.55-7.86 million versus 54.4 million) with minimal accuracy loss. On a real-world benchmark, FloodLlama-MI6 achieves 98.62% accuracy, compared with 86.61% for the published STURM-FloodDepth baseline.

洪水预测视觉语言模型可解释性精细估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。