发现学习率与权重衰减的组合不变性，大幅简化超参调优。
Efficient Hyperparameter Tuning via Trajectory Invariance Principle
- 发现损失曲线、梯度噪声等在学习率与权重衰减组合下保持不变
- 将二维超参空间压缩为一维，可沿显著方向高效调参
- 适用于大规模训练场景，尤其适合优化深度模型超参
随着大规模训练中超参数调优成本上升，高效调优方法变得至关重要。然而指导调优的原则仍不充分。本文研究了包括批量大小、学习率和权重衰减在内的广泛超参数，发现一种称为轨迹不变性的现象：预训练过程中的损失曲线、梯度噪声和梯度范数在学习率与权重衰减的组合下表现出高度重合的不变性。这一现象将原始二维超参数空间压缩为一维，从而形成高效的调优规则——沿着轨迹不变性揭示的显著方向进行调优。此外，本文还改进了先前的缩放定律，并挑战了几种现有观点。整体而言，本工作提出了新的高效调优原则，为未来缩放定律研究提供了新思路。
原文摘要 · Abstract (English)
As hyperparameter tuning becomes increasingly costly at scale, efficient tuning methods are essential. Yet principles for guiding hyperparameter tuning remain limited. In this work, we seek to establish such principles by considering a broad range of hyperparameters, including batch size, learning rate, and weight decay. We identify a phenomenon we call trajectory invariance, where pre-training loss curves, gradient noise, and gradient norm exhibit invariance--closely overlapping--with respect to a quantity that combines learning rate and weight decay. This phenomenon effectively reduces the original two-dimensional hyperparameter space to one dimension, yielding an efficient tuning rule: follow the salient direction revealed by trajectory invariance. Furthermore, we refine previous scaling laws and challenge several existing viewpoints. Overall, our work proposes new principles for efficient tuning and inspires future research on scaling laws.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。