针对气象专家的多维度评测基准，揭示模型在本地化推理上的短板。
K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology
- 基于韩国专业资格考试构建多维评测框架
- 55个模型中多数在图表理解与逻辑推理上表现差
- 本地模型优于大模型，凸显文化依赖性重要性
为提升面向韩国气象预报员的多模态大模型助手实用性，当前缺乏基于权威来源的多维度专家级评估体系。为此，我们提出K-MetBench，一个基于国家资质考试的诊断性评测基准。该基准揭示了四个关键维度的差距：专家级图表视觉推理能力、专家验证逻辑的有效性、韩语特定地理文化理解力，以及细粒度领域分析能力。对55个模型的评估显示，模型在解读专业图表方面存在显著模态鸿沟，在正确预测下仍会虚构逻辑，表现出严重幻觉。值得注意的是，韩国本地模型在本地情境中显著优于更大规模的全球模型，表明参数量扩展无法解决文化依赖问题。K-MetBench为开发可靠且具文化敏感性的专家级AI代理提供了路线图。数据集已公开于 https://huggingface.co/datasets/soyeonbot/K-MetBench。
原文摘要 · Abstract (English)
The development of practical (multimodal) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce K-MetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geo-cultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/datasets/soyeonbot/K-MetBench .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。