用上下文学习优化复杂自动机器学习流水线，提升模型选择与调优效率。
In-Context Decision Making for Optimizing Complex AutoML Pipelines
- 基于后验采样扩展为最大k臂老虎机问题，结合上下文学习建模
- 在3个基准任务上优于现有贝叶斯优化与AutoML方法
- 适用于需要多阶段调优的现代机器学习流水线，如微调与集成
组合算法选择与超参数优化（CASH）是传统自动机器学习系统的核心。然而，随着预训练模型的发展，现代机器学习流程已超越超参数优化，常需微调、集成等适配技术。尽管核心挑战——为下游任务选择最优模型仍存在，但模型流水线的异质性要求新型自动机器学习方法。本文将CASH框架扩展至选择与适配现代机器学习流水线。提出PS-PFN方法，通过将后验采样（PS）推广至最大k臂老虎机问题，利用拟合数据的神经网络（PFNs）实现上下文学习，高效估计最大值后验分布。我们进一步扩展该方法以考虑不同动作的代价差异，并为每个动作使用独立的PFN建模奖励分布。在1个新任务及2个现有标准基准任务上的实验表明，PS-PFN在性能上显著优于其他贝叶斯优化与自动机器学习策略。代码与数据已公开于https://github.com/amirbalef/CASHPlus。
原文摘要 · Abstract (English)
Combined Algorithm Selection and Hyperparameter Optimization (CASH) has been fundamental to traditional AutoML systems. However, with the advancements of pre-trained models, modern ML workflows go beyond hyperparameter optimization and often require fine-tuning, ensembling, and other adaptation techniques. While the core challenge of identifying the best-performing model for a downstream task remains, the increasing heterogeneity of ML pipelines demands novel AutoML approaches. This work extends the CASH framework to select and adapt modern ML pipelines. We propose PS-PFN to efficiently explore and exploit adapting ML pipelines by extending Posterior Sampling (PS) to the max k-armed bandit problem setup. PS-PFN leverages prior-data fitted networks (PFNs) to efficiently estimate the posterior distribution of the maximal value via in-context learning. We show how to extend this method to consider varying costs of pulling arms and to use different PFNs to model reward distributions individually per arm. Experimental results on one novel and two existing standard benchmark tasks demonstrate the superior performance of PS-PFN compared to other bandit and AutoML strategies. We make our code and data available at https://github.com/amirbalef/CASHPlus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。