用简单方法在线监测模型训练的陡峭度，自动防崩溃且无需调参。
Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam
- 利用梯度回溯法自带的步长信息，低成本估算损失曲面陡峭度。
- 在多个数据集上，步长对数与主曲率对数相关性达-0.91至-0.95。
- 可作为无需调参的自适应学习率保险，避免训练发散。
局部损失尖锐度（即海森矩阵最大特征值λ₁）决定了稳定梯度步长上限，常规测量需兰佐斯或海森-向量乘积。而一次阿米尔乔回溯线搜索已隐含该信息：接受的步长α将方向曲率限制在回溯因子设定的倍数区间内——恰好是测试步长上的平均曲率，实证表明其与$ q = g^ op H g/ \|g\|^2 $在该区间内一致。在CIFAR-10、Fashion-MNIST和Imagenette上，$\log α$与$\log λ_1$的皮尔逊相关性为-0.91至-0.95，经每轮去趋势检验后仍保持-0.60至-0.70，实现低代价在线的边缘稳定性读数。作为安全阀而非加速器使用，该读数可限制过大初始学习率。采用单一固定协议：在初始化及前50步沿Adam更新方向探测，并将学习率上限设为最小读数的两倍，可在学习率范围从10⁻³到3.0的全网格下消除所有发散，包括GPT-2预训练规模；在其他架构中仅1例边缘情况失败，开销约1%，且当上限未触发时训练比特完全相同。协议中无参数针对具体架构调整，故称校准自由。该保障仅保证不发散，不保证精度：当有效训练范围窄时，受限训练准确率显著下降（如在AG News上接近随机水平），且测量显示：若学习率上限在初始化时固定，必在预训练尺度失效——因损失曲面在前五步内迅速变尖，而常规温启动已无法弥补此差距。
原文摘要 · Abstract (English)
The local sharpness of the loss, the top Hessian eigenvalue $λ_1$, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector products. A single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step $α$ brackets the directional curvature along the probed direction within the multiplicative band set by the backtracking factor: exactly the curvature averaged over the tested step, and empirically $q = g^\top H g/\|g\|^2$ to within that band. Across CIFAR-10, Fashion-MNIST and Imagenette, $\logα$ tracks $\logλ_1$ at Pearson $-0.91$ to $-0.95$, and the relation survives a per-run detrending check at $-0.60$ to $-0.70$, a low-cost online Edge-of-Stability reading of the slow sharpness component. Used as a safeguard rather than a faster optimiser, the reading caps a too-large initial learning rate. A single fixed protocol, probing along Adam's own update direction at initialisation and over the first fifty optimiser steps and capping the rate at twice the smallest reading, removes every divergence across learning-rate grids spanning $10^{-3}$ to $3.0$ and at GPT-2 pretraining scale, and all but one marginal case across the further architectures we test, at about $1\%$ overhead, and it leaves training bit-identical whenever the cap does not bind. No constant in the protocol is tuned per architecture; this is the sense in which the safeguard is calibration-free. The guarantee is divergence, not accuracy: where the productive range is narrow the capped run survives at strongly reduced accuracy (chance level on AG News at aggressive rates), and our measurements show why any cap frozen at initialisation must fail at pretraining scale: the loss surface sharpens within the first five optimiser steps, the gap warmup has always filled by convention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。