构建地球系统动态评估基准,测试智能体在灾害分析中的科学推理能力
EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards

- 设计405个可复现任务,覆盖199个真实事件和19类灾害
- 最佳模型平均答对率84.65%,严格匹配率仅34.81%
- 适合评估科学智能体在多源异构数据下的端到端可靠性
地球系统分析需从来源、尺度、时间与模态各异的观测中重构动态物理过程。自然灾害使这项工作尤为重要,因证据不全可能改变对严重性、暴露程度与机制的判断。我们提出EarthVerse基准,通过封装式调查评估科学智能体。该基准包含405个可复现任务,基于199个已记录事件与19类灾害。智能体需检查异构事件包,选择兼容证据,执行透明计算,协调来源差异,并在最终答案中保留溯源信息。我们提供可执行的真值,将每项任务分解为细粒度答案单元,并配备任务特定评分标准,评估研究过程的同时允许多种有效路径。我们在受控工具使用协议下评估25个模型与智能体系统,通过控制实验定位证据获取、工具选择、记忆、推理、交互及科学执行等环节的失败。跨系统中,最高平均答案单元准确率为84.65%,而最高严格匹配率(Strict@95)仅为34.81%。这一差距表明当前智能体虽能完成单步操作,却难以维持跨证据、尺度、单位、计算与物理解释的一致链条。EarthVerse为动态地球系统中的端到端科学可靠性评估提供了可复现基础。
原文摘要 · Abstract (English)
Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。