提出理想化随机Polyak步长，实现最优收敛速度并用于无需调参的模型蒸馏。
Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation
- 设计理想化随机步长SPS*，仅需局部梯度有界条件即可保证收敛
- 在平滑设置下实现O(1/√t)的任意时刻收敛率，达到理论最优
- 可与动量结合，用于无须调参的GPT-2蒸馏，适合实际部署场景
我们提出了一个理想化的随机Polyak步长（SPS*）的通用收敛定理。除凸性外，仅需局部期望梯度有界条件，该条件包含局部光滑和局部Lipschitz损失的特例。SPS*被称为理想化，因其需访问每个训练批次在解处的损失值，且在全局Lipschitz函数情况下达到最优下界，是首个在平滑设置中实现任意时刻O(1/√t)收敛率的Polyak步长。我们展示了如何将SPS*与动量结合，使最终迭代点也获得相同良好速率。最后通过实验验证理论，并展示如何在无需超参数调优的情况下，将教师GPT-2模型蒸馏为更小的学生模型。
原文摘要 · Abstract (English)
We provide a general convergence theorem of an idealized stochastic Polyak step size called SPS$^*$. Besides convexity, we only assume a local expected gradient bound, that includes locally smooth and locally Lipschitz losses as special cases. We refer to SPS$^*$ as idealized because it requires access to the loss for every training batch evaluated at a solution. It is also ideal, in that it achieves the optimal lower bound for globally Lipschitz function, and is the first Polyak step size to have an $O(1/\sqrt{t})$ anytime convergence in the smooth setting. We show how to combine SPS$^*$ with momentum to achieve the same favorable rates for the last iterate. We conclude with several experiments to validate our theory, and a more practical setting showing how we can distill a teacher GPT-2 model into a smaller student model without any hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。