用大模型生成特征工程代码，提升数据建模效果
Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists
- 让大模型根据数据描述自动生成特征工程代码
- 使用XGBoost在新数据上的性能提升作为评分依据
- 低成本高效评估大模型的数据科学能力
我们提出一个基准测试，用于评估大语言模型在数据科学中最知识密集的任务之一——编写特征工程代码的能力，这需要领域知识以及对问题和数据结构的深刻理解。模型接收包含数据描述的提示，被要求生成转换数据的代码。评估分数基于在修改后的数据集上训练的XGBoost模型相比原始数据的性能提升。通过对当前最先进模型的广泛评估并与现有基准对比，我们证明所提出的FeatEng能以低成本、高效率地全面评估大模型的能力，优于现有方法。
原文摘要 · Abstract (English)
We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure. The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。