针对噪声梯度的鲁棒训练，提出按样本裁剪梯度的新方法。
Robust and Fast Training via Per-Sample Clipping
- 对每个样本的梯度分别裁剪，提升优化稳定性。
- 在CIFAR-100上训练AlexNet时，优于标准SGD和梯度裁剪。
- 梯度累积时按批次裁剪，性能提升且几乎无额外开销。
我们提出一种基于逐样本梯度裁剪的鲁棒梯度估计器，并从理论与实验两方面分析其性质。结果表明，该方法——逐样本裁剪的SGD（PS-Clip-SGD）——在重尾梯度噪声下，对非凸优化问题能达到最优期望收敛速率。此外，我们建立了高概率收敛保证，其与期望速率仅在失败概率的多项对数因子内有差异。通过多组数值实验验证，即使考虑逐样本裁剪带来的额外计算时间，PS-Clip-SGD在训练AlexNet于CIFAR-100数据集时仍优于带动量的普通SGD及标准梯度裁剪。同时，我们实证发现：在梯度累积场景中，对小批量执行裁剪可显著提升训练性能，且几乎不增加计算成本。这一发现挑战了通常在完成所有累积步骤后再进行裁剪的常规做法。
原文摘要 · Abstract (English)
We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically. We show that the resulting method, per-sample clipped SGD (PS-Clip-SGD), achieves optimal in-expectation convergence rates for non-convex optimization problems under heavy-tailed gradient noise. Moreover, we establish high-probability convergence guarantees that match the in-expectation rates up to polylogarithmic factors in the failure probability. We complement our theoretical results with multiple numerical experiments. In particular, we demonstrate that PS-Clip-SGD outperforms both vanilla SGD with momentum and standard gradient clipping when training AlexNet on the CIFAR-100 dataset, even after accounting for the additional computational time caused by per-sample clipping. We also empirically show that, in the presence of gradient accumulation, applying clipping at the mini-batch level can improve training performance while incurring virtually no additional computational cost. This finding is particularly interesting, as it contradicts the common practice of applying clipping only after all accumulation steps have been completed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。