通过梯度变化判断组件是否收敛，实现更高效的Transformer训练。
GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stopping
- 基于反向传播中梯度变化幅度，动态停止各组件更新。
- 训练速度提升1.57至7.22倍，语言任务准确率提高1.2%。
- 适合需要高效微调的大模型应用，尤其在资源受限场景。
早期停止通过监控全局验证损失来统一终止所有参数更新，但大型Transformer需大量验证推理时间，计算成本高。我们提出一种新型梯度驱动的早停方法GradES，它在Transformer组件（注意力投影和前馈层矩阵）内运行。实验发现，不同组件在语言与视觉-语言模型微调中收敛速度各异。GradES追踪这些矩阵在训练过程中的梯度变化幅度。当某投影矩阵的梯度变化幅度低于收敛阈值τ，即单独停止该矩阵的更新，从而避免昂贵的验证步骤，同时允许收敛慢的矩阵继续学习。该方法使训练时间提速1.57–7.22倍，并通过提前防止过拟合提升泛化能力，在语言任务上平均准确率提高1.2%，多模态基准上提升3.88%。
原文摘要 · Abstract (English)
Early stopping monitors global validation loss and halts all parameter updates simultaneously, which is computationally costly for large transformers due to the extended time required for validation inference. We propose \textit{GradES}, a novel gradient-based early stopping approach that operates within transformer components (attention projections and Feed-Forward layer matrices). We found that different components converge at varying rates during fine-tuning for both language and vision-language models. \textit{GradES} tracks the magnitude of gradient changes in backpropagation for these matrices during training. When a projection matrix's magnitude of gradient changes fall below a convergence threshold $τ$, we exclude that projection matrix from further updates individually, eliminating costly validation passes while allowing slow converging matrices to continue learning. \textit{GradES} speeds up training time by 1.57--7.22$\times$ while simultaneously enhancing generalization through early prevention of overfitting, resulting in 1.2\% higher average accuracy in language tasks and 3.88\% on multimodal benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。