构建气象预报文本生成评测数据集与框架，验证大模型对天气现象的描述能力。
SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future
- 构建包含136万条气象预报文本与多源气象图像的SynopticBench数据集。
- 提出SPACE评估框架，可有效衡量模型对天气系统描述的准确性与覆盖度。
- 揭示现有评测指标在气象文本生成任务中的局限性，助力气候智能生成研究。
视觉语言模型（VLMs）在图像描述、报告生成等多模态任务中取得显著进展，但基于气象数据生成文本极具挑战性，因大气系统具有混沌性，且在多时空尺度下快速变化。为可靠量化现有VLM在气象预报数据上的表现，本文提出SynopticBench，一个高质量数据集，包含美国大陆地区国家气象局生成的1,367,041条区域预报讨论文本，对应500mb位势高度、2米温度和850mb风速图像。同时提出合成现象对齐与覆盖评估（SPACE）框架，可有效评估文本对天气系统现象的描述质量。在主流VLM上进行的大量实验揭示了现有评估指标在此领域的敏感性，推动了对天气与气候文本生成的进一步探索。
原文摘要 · Abstract (English)
Recent advances in visual-language models (VLMs) have led to significant improvements in a plethora of complex multimodal tasks like image captioning, report generation, and visual perception. However, generating text from meteorological data is highly challenging because the atmosphere is a chaotic system that is rapidly changing at various spatial and temporal scales. Given the complexity of atmospheric phenomena, it is critical to verifiably quantify the effectiveness of existing VLMs on weather forecasting data. In this work, we present SynopticBench, a high-quality dataset consisting of 1,367,041 text samples of Area Forecast Discussions created by the National Weather Service over the continental United States paired to images of 500mb geopotential height, 2 meter temperature, and 850mb wind velocity in weather forecasts. We also present Synoptic Phenomena Alignment and Coverage Evaluation (SPACE), a novel evaluation framework that can be used to effectively estimate the quality of text descriptions of synoptic weather phenomena. Extensive experiments on generating forecast discussions using state-of-the-art VLMs show the sensitivity of existing evaluation metrics in this domain and enable further exploration into synoptic weather and climate text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。