提出AlphaAdam，通过动态掩码加速大模型训练收敛并提升稳定性。
AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates
- 按层内参数历史动量与梯度方向一致性生成动态掩码。
- 在GPT-2、RoBERTa和Llama-7B上收敛更快，计算效率更高。
- 适用于多数基于动量的优化器，适合大模型高效训练场景。
大语言模型训练中，如何更高效稳定地更新参数始终是关键挑战。现有方法通常通过低维分解或逐层选择性更新，实现接近全参数更新的性能。本文提出AlphaAdam，从层内参数更新角度出发，通过解耦参数更新并动态调节更新强度，加速收敛并提升训练稳定性。基于历史动量与梯度方向的一致性构建参数掩码，并结合自适应掩码强度策略，在保证理论收敛性的同时实现高效优化，可适配大多数基于动量的优化器。大量实验表明，AlphaAdam在收敛速度和计算效率上均优于AdamW等先进方法，涵盖GPT-2预训练及微调任务、RoBERTa与Llama-7B。该方法通过层内异步掩码自适应更新，实现了大模型优化器增强框架。代码已开源：https://github.com/MaeChd/AlphaAdam。
原文摘要 · Abstract (English)
In the training of large language models (LLMs), updating parameters more efficiently and stably has always been an important challenge. To achieve efficient parameter updates, existing methods usually achieve performance comparable to full parameter updates through methods such as low-dimensional decomposition or layer-wise selective updates. In this work, we propose AlphaAdam, an optimization framework for LLM from the perspective of intra-layer parameter updates. By decoupling parameter updates and dynamically adjusting their strength, AlphaAdam accelerates convergence and improves training stability. We construct parameter masks based on the consistency of historical momentum and gradient direction and combine them with an adaptive mask strength strategy to ensure efficient optimization and theoretical convergence guarantees, which is also applicable to most momentum-based optimizers. Extensive experiments show that AlphaAdam outperforms state-of-the-art methods such as AdamW in terms of convergence speed and computational efficiency across tasks, including GPT-2 pre-trained and fine-tuned RoBERTa and Llama-7B. Our AlphaAdam implements an optimizer enhancement framework for LLMs through intra-layer asynchronous masked adaptive updates. Our code is available in this https://github.com/MaeChd/AlphaAdam.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。