评测视觉语言模型在恶劣天气下的推理分割能力,发现性能随天气恶化而下降。
WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models
- 用合成天气数据构建可控测试集,模拟不同严重程度的雨雪雾
- 真实天气数据集通过大模型提示生成语义一致的问答对
- 覆盖5类推理维度,揭示不同天气导致不同脆弱性模式
现有视觉语言模型(VLMs)在基于推理的分割任务上表现优异,但当前基准大多基于理想条件下拍摄的高质量图像。这引发关键问题:当雨、雪、雾等恶劣天气严重削弱视觉线索时,VLM能否保持可靠的推理分割能力?为此,我们提出WeatherReasonSeg,一个评估VLM在恶劣天气下推理分割性能的基准。它包含两个互补组件:其一,通过在现有分割数据集上施加不同程度的合成天气,构建可控制的推理数据集,实现细粒度鲁棒性分析;其二,为捕捉真实世界复杂性,我们收集了一个真实恶劣天气推理分割数据集,使用掩码引导的大语言模型生成语义一致的查询。我们进一步在五个推理维度(功能、应用场景、结构属性、交互关系、需求匹配)上扩展评估范围。对多种VLM的大量实验揭示两个关键发现:(1) VLM性能随天气严重程度单调下降;(2) 不同天气类型引发不同的脆弱性模式。我们希望WeatherReasonSeg能成为推动鲁棒、天气感知推理发展的基础。
原文摘要 · Abstract (English)
Existing vision-language models (VLMs) have demonstrated impressive performance in reasoning-based segmentation. However, current benchmarks are primarily constructed from high-quality images captured under idealized conditions. This raises a critical question: when visual cues are severely degraded by adverse weather conditions such as rain, snow, or fog, can VLMs sustain reliable reasoning segmentation capabilities? In response to this challenge, we introduce WeatherReasonSeg, a benchmark designed to evaluate VLM performance in reasoning-based segmentation under adverse weather conditions. It consists of two complementary components. First, we construct a controllable reasoning dataset by applying synthetic weather with varying severity levels to existing segmentation datasets, enabling fine-grained robustness analysis. Second, to capture real-world complexity, we curate a real-world adverse-weather reasoning segmentation dataset with semantically consistent queries generated via mask-guided LLM prompting. We further broaden the evaluation scope across five reasoning dimensions, including functionality, application scenarios, structural attributes, interactions, and requirement matching. Extensive experiments across diverse VLMs reveal two key findings: (1) VLM performance degrades monotonically with increasing weather severity, and (2) different weather types induce distinct vulnerability patterns. We hope WeatherReasonSeg will serve as a foundation for advancing robust, weather-aware reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。