首个专注地理空间代码生成的LLM,提升科研效率
GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks
- 基于Code Llama-7B微调,融合领域语料与高效参数训练
- 多项指标超越基线模型,最高提升32.1%
- 适合地理、气候等领域的科研人员快速生成分析代码
地学领域对时空数据与建模任务的需求日益增长,地理空间代码生成技术成为提升生产力的关键。尽管大语言模型(LLMs)在代码生成方面展现潜力,但在地理空间任务中常因缺乏领域知识和代码语料而出现拒绝生成或幻觉问题。本文提出并开源了GeoCode-PT与GeoCode-SFT语料库,以及GeoCode-Eval评估数据集。通过使用QLoRA与LoRA进行预训练与微调,构建了首个专注于地理空间代码生成的70亿参数大模型GeoCode-GPT-7B,其基于Code Llama-7B。此外,建立了包含选项匹配、专家验证与提示工程评分的综合评估框架,并在GeoCode-Eval数据集上系统评估了该模型。实验表明,GeoCode-GPT在多选题准确率上比其他模型高出9.1%至32.1%,代码摘要能力提升1.7%至25.4%,代码生成能力提升1.2%至25.1%。本工作为提升大模型在地理空间代码生成中的表现提供了可行方案与实证支持,拓展了领域专用模型的应用边界,也为释放其在地理空间领域的潜力提供了重要参考。
原文摘要 · Abstract (English)
The increasing demand for spatiotemporal data and modeling tasks in geosciences has made geospatial code generation technology a critical factor in enhancing productivity. Although large language models (LLMs) have demonstrated potential in code generation tasks, they often encounter issues such as refusal to code or hallucination in geospatial code generation due to a lack of domain-specific knowledge and code corpora. To address these challenges, this paper presents and open-sources the GeoCode-PT and GeoCode-SFT corpora, along with the GeoCode-Eval evaluation dataset. Additionally, by leveraging QLoRA and LoRA for pretraining and fine-tuning, we introduce GeoCode-GPT-7B, the first LLM focused on geospatial code generation, fine-tuned from Code Llama-7B. Furthermore, we establish a comprehensive geospatial code evaluation framework, incorporating option matching, expert validation, and prompt engineering scoring for LLMs, and systematically evaluate GeoCode-GPT-7B using the GeoCode-Eval dataset. Experimental results show that GeoCode-GPT outperforms other models in multiple-choice accuracy by 9.1% to 32.1%, in code summarization ability by 1.7% to 25.4%, and in code generation capability by 1.2% to 25.1%. This paper provides a solution and empirical validation for enhancing LLMs' performance in geospatial code generation, extends the boundaries of domain-specific model applications, and offers valuable insights into unlocking their potential in geospatial code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。