让优化器自动调节不同参数组的学习率和权重衰减,提升训练效率与性能。
A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay

- 用轻量Transformer根据梯度等统计特征动态调整各组学习率和权重衰减。
- 在5个任务上表现优于AdamW,最高提速17.11%或提效11.08%。
- 支持领域知识注入,适合需要精细调参的复杂模型训练。
自适应优化器如AdamW对所有参数组使用统一超参数,忽视了不同层与模块间优化动态的异质性。本文提出MetaAdamW,通过引入自注意力机制,动态调节各参数组的学习率和权重衰减。该调节因子由一个轻量级Transformer编码器生成,其输入为各参数组的统计特征(梯度范数、动量范数、相关性)。为训练注意力模块,设计了结合梯度对齐、损失下降与泛化差距的元学习目标。核心创新在于将同方差不确定性加权(HUW)扩展为带任务优先级的正则项缩放机制,使领域知识可直接指导自动损失平衡。在五类任务(时间序列预测ETT、语言建模WikiText-2、机器翻译Multi30k、图像分类CIFAR-10、情感分析IMDB)上的实验表明,MetaAdamW在验证损失、准确率或困惑度上均持续优于标准AdamW基线;根据任务不同,可减少训练时间(最多17.11%)或提升性能(最多11.08%),且开销适中;部分情况下还可缓解因过早早停导致的收敛不足问题。消融实验验证了各组件的有效性,包括特征版本、分组策略及所提出的优先级注入不确定性加权。
原文摘要 · Abstract (English)
Adaptive optimizers like AdamW apply uniform hyperparameters across all parameter groups, ignoring heterogeneous optimization dynamics across layers and modules. We address this limitation by proposing MetaAdamW - a new optimizer that integrates a self-attention mechanism to dynamically modulate per-group learning rates and weight decay. The modulation factors are produced by a lightweight Transformer encoder that operates on statistical features (gradient norms, momentum norms, correlations) extracted from each parameter group. To train the attention module, we introduce a meta-learning objective that combines gradient alignment, loss decrease, and generalization gap. A key novel contribution is the extension of homoscedastic uncertainty weighting (HUW) with task-specific priorities that directly scale the regularization terms - enabling domain knowledge to guide automatic loss balancing. Extensive experiments on five diverse tasks-time series forecasting (ETT), language modeling (WikiText-2), machine translation (Multi30k), image classification (CIFAR-10), and sentiment analysis (IMDB) - demonstrate that MetaAdamW consistently outperforms the standard AdamW baseline in terms of validation loss, accuracy, or perplexity. Depending on the task, MetaAdamW either reduces overall training time (by up to 17.11%) or improves performance (by up to 11.08%) while introducing only moderate overhead; in some cases, it can also mitigate issues of insufficient convergence caused by premature early stopping. Ablation studies validate the effectiveness of each component, including feature versions, grouping strategies, and the proposed priority-injected uncertainty weighting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。