通过缓存优化的贝叶斯优化，显著降低大模型调参成本
Reducing Hyperparameter Tuning Costs in ML, Vision and Language Model Training Pipelines via Memoization-Awareness
- 结合管道缓存与记忆化感知的贝叶斯优化算法
- 相同时间下多生成103%候选参数，性能提升高出108%
- 适合高成本大模型训练的团队快速调参
机器学习、视觉和语言模型的训练或微调通常以流水线形式进行：包括数据准备、模型训练和评估等多个阶段。本文利用流水线结构，降低模型训练/微调中超参数调优的成本，尤其对高成本的语言模型而言意义重大。我们提出一种‘记忆化感知’的贝叶斯优化算法EEIPU，与流水线缓存系统协同工作，可在相同GPU天数内评估更多超参数候选。在机器学习（模型集成）、视觉（卷积架构）和语言（T5架构）流水线上的基准测试显示，相较于近期贝叶斯优化算法，EEIPU在相同预算下平均多生成103%的超参数候选，且验证指标平均提升幅度比其他算法高出108%（从预热迭代结束后开始计算）。
原文摘要 · Abstract (English)
The training or fine-tuning of machine learning, vision, and language models is often implemented as a pipeline: a sequence of stages encompassing data preparation, model training and evaluation. In this paper, we exploit pipeline structures to reduce the cost of hyperparameter tuning for model training/fine-tuning, which is particularly valuable for language models given their high costs in GPU-days. We propose a "memoization-aware" Bayesian Optimization (BO) algorithm, EEIPU, that works in tandem with a pipeline caching system, allowing it to evaluate significantly more hyperparameter candidates per GPU-day than other tuning algorithms. The result is better-quality hyperparameters in the same amount of search time, or equivalently, reduced search time to reach the same hyperparameter quality. In our benchmarks on machine learning (model ensembles), vision (convolutional architecture) and language (T5 architecture) pipelines, we compare EEIPU against recent BO algorithms: EEIPU produces an average of $103\%$ more hyperparameter candidates (within the same budget), and increases the validation metric by an average of $108\%$ more than other algorithms (where the increase is measured starting from the end of warm-up iterations).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。