LLM做超参优化其实不如默认配置,几轮后就被传统方法超越。
When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model
- 用默认配置启动搜索,比调用LLM还有效。
- LLM提案仅提升0.40个百分点准确率,且对测试集无改善。
- 经典方法在12轮后反超,40轮时领先0.6至0.8个百分点。
我们采用预算匹配、多种子的实验协议,在八个PMLB表格数据集上对比了基于LLM的超参优化器(LLM-OptFlow)与四种经典基线方法(随机搜索、Optuna-TPE、高斯过程贝叶斯优化、相继减半法)。在共享搜索空间下,通过成对检验和置信区间分析40个(任务, 种子)组合。结果表明:所谓强起点并非来自LLM,而是预设的默认配置——其平均最佳交叉验证准确率达88.7%,与所有七种LLM模型表现一致(差异<0.01个百分点)。LLM实际建议仅带来+0.40个百分点提升,且对保留测试集无增益(LLM-Default = -0.01 pp, p=0.92)。若将相同默认配置赋予经典方法,其优势消失:相比种子随机搜索,仅在2次评估中领先0.20个百分点,5次持平,12次落后0.37个百分点。无种子时,经典方法在12次评估内与之持平,40次时领先0.6至0.8个百分点(p≤1e-4)。仅两项行为持续存在:单任务探索失败(车辆数据),以及规则过滤机制剔除约33%无效计算但不改变精度。结论为:在表格数据超参优化中,应以合理默认值初始化经典搜索;LLM并未带来可测量的泛化优势,且在少数评估后即被超越。代码与复现脚本已公开。
原文摘要 · Abstract (English)
Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that "warm-start" search from prior knowledge, proposing strong configurations in very few evaluations. We test that claim under a budget-matched, multi-seed protocol on eight PMLB tabular benchmarks, comparing an LLM advisor (LLM-OptFlow) against four classical baselines (random search, Optuna-TPE, Gaussian-process Bayesian optimization, and successive halving) over one shared search space, with paired tests and bootstrap 95% CIs across 8 x 5 = 40 (task, seed) units. The finding is cautionary. The advisor's strong first point is not an LLM output at all: like prior LLM-HPO systems the loop is seeded with a fixed default configuration, evaluated before any model call, which alone reaches 88.7% mean best-CV, identical to within 0.01 pp across all seven advisor models tested. The LLM's own proposals add only +0.40 pp of cross-validation accuracy over that seed and nothing on held-out test (LLM-Default = -0.01 pp, p = 0.92). When the same seed is granted to classical search, the apparent lead collapses: against seeded random search it leads by +0.20 pp at 2 evaluations, is tied by 5, and is behind by 12 (-0.37 pp). Without the seed, classical search ties the advisor by 12 evaluations and beats it by 40 (+0.6 to +0.8 pp, p <= 1e-4). Two LLM-specific behaviors survive: a single-task exploration failure (vehicle), and a rule-based confidence filter that removes ~33% of wasted compute without changing accuracy. The recommendation is deflationary: on tabular HPO, seed classical search with a sensible default; an LLM advisor adds no measurable generalization benefit and is overtaken within a handful of evaluations. We release the harness and a script that reproduces every statistic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。