首个自动评估地理空间代码生成的多模态框架，提升AI写地理代码的可靠性。
AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models
- 用LLM构建端到端自动评估流程，支持函数调用与执行验证。
- 覆盖1325个测试用例，涵盖26种地球引擎数据类型，可量化分析准确率和资源消耗。
- 适合研究地理信息智能生成的学者及开发者，推动自动化代码翻译发展。
地理空间代码生成正成为人工智能与地学分析融合的关键方向，但该领域缺乏标准化的自动评估工具。为此，我们提出AutoGEEval，首个基于大语言模型（LLMs）的多模态、单元级自动化评估框架，专用于谷歌地球引擎（GEE）平台上的地理空间代码生成任务。依托GEE Python API，AutoGEEval建立基准测试集（AutoGEEval-Bench），包含1325个测试用例，覆盖26种GEE数据类型。框架集成问题生成与答案验证模块，实现从函数调用到执行验证的全流程自动化评估。支持对模型输出在准确性、资源消耗、执行效率及错误类型等方面的多维度定量分析。我们评估了18个主流大模型——包括通用型、推理增强型、代码导向型和地学专用型模型——揭示其在GEE代码生成中的性能特征与优化路径。本工作为地理空间代码生成模型的研发与评估提供统一协议与基础资源，推动自然语言到领域特定代码自动转换的前沿发展。
原文摘要 · Abstract (English)
Geospatial code generation is emerging as a key direction in the integration of artificial intelligence and geoscientific analysis. However, there remains a lack of standardized tools for automatic evaluation in this domain. To address this gap, we propose AutoGEEval, the first multimodal, unit-level automated evaluation framework for geospatial code generation tasks on the Google Earth Engine (GEE) platform powered by large language models (LLMs). Built upon the GEE Python API, AutoGEEval establishes a benchmark suite (AutoGEEval-Bench) comprising 1325 test cases that span 26 GEE data types. The framework integrates both question generation and answer verification components to enable an end-to-end automated evaluation pipeline-from function invocation to execution validation. AutoGEEval supports multidimensional quantitative analysis of model outputs in terms of accuracy, resource consumption, execution efficiency, and error types. We evaluate 18 state-of-the-art LLMs-including general-purpose, reasoning-augmented, code-centric, and geoscience-specialized models-revealing their performance characteristics and potential optimization pathways in GEE code generation. This work provides a unified protocol and foundational resource for the development and assessment of geospatial code generation models, advancing the frontier of automated natural language to domain-specific code translation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。