用树结构融合大模型与贝叶斯优化，高效解决算法与超参联合调优问题
Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH
- 构建基于蒙特卡洛树搜索的共享状态，统一算法选择与超参优化流程
- 在104个AMLB数据集上优于传统贝叶斯优化和大模型基线方法
- 动态切换大模型与贝叶斯优化主导权，适应不同搜索阶段需求
为降低机器学习的专家门槛，自动化机器学习（AutoML）领域聚焦于CASH问题，即联合自动化算法选择与超参数调优。传统贝叶斯优化（BO）存在冷启动问题，而大语言模型（LLM）可通过语义先验缓解此问题。然而，现有基于LLM的优化器在高维、结构化CASH空间中泛化能力差。本文提出LB-MCTS，一种轨迹结构化的优化框架，以蒙特卡洛树搜索（MCTS）作为算法选择、超参精调与BO-LLM提议者协同的共享状态。在该共享状态中，BO为算法提供特定的代理模型以进行定量搜索，而LLM则利用路径感知的选择性记忆生成语义提议与反思。随着代理模型性能提升，可靠性感知的提议策略在统一搜索轨迹中自适应地从LLM主导转向BO主导。在104个AMLB数据集上的实验表明，LB-MCTS始终优于基于BO、基于LLM及混合基线方法。
原文摘要 · Abstract (English)
To lower the expertise barrier in machine learning, the AutoML community has focused on the CASH problem, which jointly automates algorithm selection and hyperparameter tuning. While traditional methods like Bayesian Optimization (BO) struggle with cold-start issues, Large Language Models (LLMs) can mitigate these through semantic priors. However, existing LLM-based optimizers generalize poorly to high-dimensional, structured CASH spaces. In this paper, we propose LB-MCTS, a trajectory-structured optimization framework that uses a Monte Carlo Tree Search tree as a shared state for algorithm selection, hyperparameter refinement, and BO-LLM proposer synergy. Within this shared state, BO provides algorithm-specific surrogate modeling for quantitative search, while the LLM exploits path-aware selective memory to generate semantic proposals and reflections. As the surrogate model improves, a reliability-aware proposer policy adaptively shifts from LLM-driven to BO-driven proposals within a unified search trajectory. Experiments on 104 AMLB datasets demonstrate that LB-MCTS consistently outperforms BO-based, LLM-based, and hybrid baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。