用大模型注入领域知识，加速特征工程演化过程。
Embedding Domain-Specific Knowledge from LLMs into the Feature Engineering Pipeline
- 让大模型先生成特征名，为演化算法提供初始知识引导。
- 在3个数据集上提升测试性能，仅1个案例性能下降。
- 适合需高效特征工程且数据敏感的场景，兼容隐私保护。
特征工程是机器学习流程中不可或缺的环节，以获得稳健模型。尽管进化计算在特征选择与构造方面表现优异，但其计算成本高，源于大量评估以训练最终模型。部分原因是这些算法缺乏领域知识，导致演化过程中大量随机尝试。本文提出利用大语言模型（LLMs）作为特征构造的初始步骤，向数据集注入领域知识，从而加快演化收敛速度，节省计算资源。该方法仅向LLM提供数据集特征名称和目标变量，适用于含私有数据的场景。实验表明，在1/3的数据集（CSS、PM、IM10）上测试性能持续提升，仅在77个测试案例中有1例性能下降。此外，本文引入M6GP特征工程算法用于符号回归，可提升随机森林回归器性能，并达到与前代算法M3GP相当的竞争力。
原文摘要 · Abstract (English)
Feature engineering is mandatory in the machine learning pipeline to obtain robust models. While evolutionary computation is well-known for its great results both in feature selection and feature construction, its methods are computationally expensive due to the large number of evaluations required to induce the final model. Part of the reason why these algorithms require a large number of evaluations is their lack of domain-specific knowledge, resulting in a lot of random guessing during evolution. In this work, we propose using Large Language Models (LLMs) as an initial feature construction step to add knowledge to the dataset. By doing so, our results show that the evolution can converge faster, saving us computational resources. The proposed approach only provides the names of the features in the dataset and the target objective to the LLM, making it usable even when working with datasets containing private data. While consistent improvements to test performance were only observed for one-third of the datasets (CSS, PM, and IM10), possibly due to problems being easily explored by LLMs, this approach only decreased the model performance in 1/77 test cases. Additionally, this work introduces the M6GP feature engineering algorithm to symbolic regression, showing it can improve the results of the random forest regressor and produce competitive results with its predecessor, M3GP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。