提出GQA-μP，实现分组查询注意力的参数化迁移，大幅降低大模型调优成本。
GQA-μP: The maximal parameterization update for grouped query attention
- 从谱范数出发重新定义特征学习，推导出完整深度与权重衰减缩放规律。
- 针对非满秩权重矩阵改进谱范数，首次实现分组查询注意力的μP缩放推导。
- 验证了学习率和权重衰减在不同重复次数间的迁移效果，理论指导实践。
跨模型架构的超参数迁移可显著降低大语言模型调优所需的计算量。最大更新参数化（μP）通过严谨的数学分析实现迁移，但对新架构的推导往往困难。基于Yang等人（2023a）的谱特征学习视角，本文做出两项改进：首先，将权重的谱范数条件从启发式提升为特征学习的定义，无需依赖懒惰学习即可导出完整深度与权重衰减的缩放关系；其次，引入一种修正的谱范数，可在权重矩阵非满秩时仍保持网络权重的有效缩放规律。这使我们首次成功推导出分组查询注意力（GQA）的μP缩放规则。实验验证了学习率在不同GQA重复次数间、以及权重衰减之间的迁移有效性，充分证明理论推导的实用性。
原文摘要 · Abstract (English)
Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization (μP) ensures transfer through principled mathematical analysis but can be challenging to derive for new model architectures. Building on the spectral feature-learning view of Yang et al. (2023a), we make two advances. First, we promote spectral norm conditions on the weights from a heuristic to the definition of feature learning, and as a consequence arrive at the Complete-P depth and weight-decay scalings without recourse to lazy-learning. Second, we consider a modified spectral norm that preserves the valid scaling law of network weights when weight matrices are not full rank. This enables (to our knowledge, the first) derivation of μP scalings for grouped-query attention (GQA). We demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter as well as experiments regarding transfer over weight decay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。