构建首个Excel公式修复基准数据集,助力大模型自动纠错。
Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs
- 用少量种子样本+LLM生成618个高质量修复数据。
- 结合执行验证与大模型评判,确保数据正确性。
- 适合研究代码修复、低资源语言建模的学者使用。
Excel虽广泛使用,但对新手而言仍具挑战性,因逻辑错误或函数误用导致的运行时错误尤为常见。尽管大语言模型(LLMs)可解释错误,但自动修复语义级错误仍属开放问题。主要瓶颈在于缺乏高质量、全面的训练与评估数据。本文提出一种新型数据生成方法,构建专用于Excel公式修复的基准数据集。基于在线论坛的少量精选样本,通过少样本提示与LLM合成扩展,结合LLM-as-a-Judge验证框架及执行检查,确保生成数据的正确性与语义保真度。最终产出618个高质样本,覆盖常见运行时错误。此外,提出一种上下文感知的基线修复方法,利用LLM融合故障公式与表格上下文信息。在新数据集上评估GPT-4o、GPT-4.1、Phi-3、Mistral等模型性能,采用执行级指标。人工标注验证数据集质量,并揭示错误与函数分布特征。该生成方法高度可扩展,可迁移至其他低资源编程语言的代码修复任务。
原文摘要 · Abstract (English)
Excel is a pervasive yet often complex tool, particularly for novice users, where runtime errors arising from logical mistakes or misinterpretations of functions pose a significant challenge. While large language models (LLMs) offer promising assistance by explaining formula errors, the automated correction of these semantic runtime errors remains an open problem. A primary challenge to advancing models for such scenarios is the severe lack of high-quality, comprehensive datasets for training and rigorous evaluation. This paper addresses this gap by introducing a novel approach for constructing a benchmark dataset specifically designed for Excel formula repair. We propose a data generation pipeline, which leverages a small set of curated seed samples from online forums to synthetically expand the dataset. Our pipeline integrates few-shot prompting with LLMs and employs a robust \textit{LLM-as-a-Judge} validation framework, combined with execution-based checks to ensure the correctness and semantic fidelity of the generated data. This process produced a benchmark dataset of 618 high-quality samples, covering common runtime errors. Furthermore, we propose a context-aware baseline technique for Excel formula repair that utilizes LLMs to leverage both the faulty formula, and relevant spreadsheet context. We evaluate the performance of various LLMs (GPT-4o, GPT-4.1, Phi-3, Mistral) on our newly generated benchmark using execution-based metrics. Our analysis demonstrates the dataset's quality through manual annotation and provides insights into error and function distributions. The proposed generation methodology is highly scalable and can be readily adapted to create evaluation benchmarks for similar code repair tasks in other low-resource programming languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。