用冻结层数量做低精度评估，节省显存与计算。
Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
- 以训练时冻结的层数作为新的低精度评估方式。
- 在ResNet和Transformer上实现显著显存与计算节省。
- 适合资源受限场景下的高效超参优化，尤其适合硬件资源受限者。
随着模型规模扩大，高效且低成本的超参数优化（HPO）方法对深度学习流程愈发关键。尽管多保真度超参数优化（MF-HPO）通过使用低保真度估计来减少深度学习训练的计算资源消耗，但现有保真度来源在计算和内存约束下表现不佳。本文提出一种新型保真度来源：训练过程中被冻结的层数。对于深层网络，该方法在保持低保真度下超参数间排名相关性的同时，显著降低计算与内存开销。我们在ResNets和Transformers上进行了实证评估，并进一步分析了冻结层数作为保真度在使用GPU资源作为保真度时的有效性，以及与其他保真度来源结合进行联合多保真度优化的潜力。该工作为将硬件资源作为保真度开辟新应用，并为探索联合保真度空间的优化算法提供新机遇。
原文摘要 · Abstract (English)
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity HPO (MF-HPO) trades off computational resources required for DL training with lower fidelity estimations, existing fidelity sources often fail under lower compute and memory constraints. We propose a novel fidelity source: the number of layers that are trained or frozen during training. For deep networks, this approach offers significant compute and memory savings while preserving rank correlations between hyperparameters at low fidelities compared to full model training. We demonstrate this in our empirical evaluation across ResNets and Transformers and additionally analyze the utility of frozen layers as a fidelity in using GPU resources as a fidelity in HPO, and for a combined MF-HPO with other fidelity sources. This contribution opens new applications for MF-HPO with hardware resources as a fidelity and creates opportunities for improved algorithms navigating joint fidelity spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。