用大模型与贝叶斯优化协同优化特征工程，提升自动机器学习效果
CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization
- 大模型结合思维树生成灵活特征工程路径
- 贝叶斯优化同步调优下游模型超参数
- 双模块互条件决策，适合复杂数据建模场景
特征工程（FE）在自动化机器学习（AutoML）中至关重要，但传统方法受限于固定搜索空间且缺乏领域感知。尽管大语言模型（LLM）能通过语义推理生成无限特征算子，现有方法仅聚焦孤立任务如特征生成，未能构建自由形式的特征工程流水线。此外，它们很少与下游模型的超参数优化（HPO）结合，导致采用“先特征工程再优化”的贪心流程，无法捕捉强交互作用。本文提出CoFEH框架，通过协同方式交替执行基于大模型的特征工程与贝叶斯优化（BO）。该框架利用树状思维（TOT）驱动的特征优化器探索灵活流水线，集成贝叶斯优化模块进行超参数调优，并引入动态选择器自适应切换优化步骤。关键创新在于双向条件机制，使大模型与贝叶斯优化共享上下文，实现相互启发。实验表明，在独立特征工程和联合特征工程+超参数优化两种设置下，CoFEH均显著优于传统及基于大模型的基线方法。
原文摘要 · Abstract (English)
Feature Engineering (FE) is pivotal in automated machine learning (AutoML) but remains a bottleneck for traditional methods, which operate within rigid search spaces and lack domain awareness. While Large Language Models (LLMs) offer a promising alternative to generate unbounded operators with semantic reasoning, existing methods focus on isolated subtasks such as feature generation, falling short of free-form FE pipelines. Moreover, they are rarely coupled with hyperparameter optimization (HPO) of the downstream ML model, leading to greedy "FE-then-HPO" workflows that cannot capture strong FE-HPO interactions. In this paper, we present CoFEH, a collaborative framework that interleaves LLM-based FE and Bayesian HPO for robust end-to-end AutoML. CoFEH uses an LLM-driven FE optimizer powered by Tree of Thought (TOT) to explore flexible FE pipelines, a Bayesian optimization (BO) module to solve HPO, and a dynamic optimizer selector that adaptively interleaves FE and HPO steps. Crucially, we introduce a mutual conditioning mechanism that shares context between LLM and BO, enabling mutually informed decisions. Experiments show that CoFEH outperforms both traditional and LLM-based baselines in both standalone FE and joint FE+HPO settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。