测试大模型在地球观测任务中的可靠性，发现代码失败率超五成
Towards LLM Agents for Earth Observation
- 构建140个来自NASA文章的真假问题基准，覆盖13主题17卫星传感器
- 主流大模型调用地球引擎时代码失败率超58%，准确率仅33%
- 通过合成数据微调降低失败率，小模型（Llama-3.1-8B）可追平大模型表现
地球观测（EO）为环境监测、灾害管理、气候科学等提供关键行星数据。本文探讨：人工智能系统是否已具备可靠执行地球观测的能力？我们提出了 extit{datasetname}，一个包含140个来自NASA地球观测文章的真假问题基准，覆盖13个主题和17种卫星传感器。利用谷歌地球引擎API作为工具，大模型代理仅实现33%的准确率，因代码在超过58%的情况下无法运行。通过微调合成数据，我们显著降低了开源模型的失败率，使更小的模型（如Llama-3.1-8B）达到与更大模型（如DeepSeek-R1）相当的准确率。研究揭示了当前AI代理在自动化地球观测中面临的关键挑战，并指明改进路径。项目主页见 https://iandrover.github.io/UnivEarth。
原文摘要 · Abstract (English)
Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. Here we ask: Are AI systems ready for reliable Earth Observation? We introduce \datasetnamenospace, a benchmark of 140 yes/no questions from NASA Earth Observatory articles across 13 topics and 17 satellite sensors. Using Google Earth Engine API as a tool, LLM agents can only achieve an accuracy of 33% because the code fails to run over 58% of the time. We improve the failure rate for open models by fine-tuning synthetic data, allowing much smaller models (Llama-3.1-8B) to achieve comparable accuracy to much larger ones (e.g., DeepSeek-R1). Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward. The project page is available at https://iandrover.github.io/UnivEarth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。