提出新优化器TAM，让模型训练更稳更快。
Torque-Aware Momentum
- 根据梯度与动量夹角动态调整阻尼，抑制震荡
- 在图像分类和大模型微调中提升泛化性能
- 可兼容SGD和Adam，适合复杂任务训练
高效探索复杂的损失曲面是深度神经网络性能的关键。尽管基于动量的优化器广泛应用于前沿模型，但经典动量在面对大而错位的梯度时仍易引发振荡。为此，我们提出扭矩感知动量（Torque-Aware Momentum, TAM），其引入基于新梯度与前一动量夹角的阻尼因子，稳定训练过程中的更新方向。实验表明，TAM 可与 SGD 和 Adam 结合使用，在图像分类与大语言模型微调等多种任务中，相比经典动量优化器，显著增强探索能力，更好应对分布偏移，并提升泛化性能。
原文摘要 · Abstract (English)
Efficiently exploring complex loss landscapes is key to the performance of deep neural networks. While momentum-based optimizers are widely used in state-of-the-art setups, classical momentum can still struggle with large, misaligned gradients, leading to oscillations. To address this, we propose Torque-Aware Momentum (TAM), which introduces a damping factor based on the angle between the new gradients and previous momentum, stabilizing the update direction during training. Empirical results show that TAM, which can be combined with both SGD and Adam, enhances exploration, handles distribution shifts more effectively, and improves generalization performance across various tasks, including image classification and large language model fine-tuning, when compared to classical momentum-based optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。