通过预测权重并验证，加速大模型训练，适配不同学习阶段。
Leap+Verify: Regime-Adaptive Speculative Weight Prediction for Accelerating Neural Network Training
- 按激活相似度划分训练阶段，分阶段预测未来权重。
- 小模型在稳定期可实现24%预测接受率，大模型在过渡期达37%。
- 适合追求训练加速的大模型研究者，尤其关注效率优化者。
我们提出Leap+Verify框架，利用推测执行思想——预先预测未来模型权重，并在确认前验证其有效性——以加速神经网络训练。受语言模型推理中推测解码及自动可扩展计算（ASC）架构启发，该框架通过激活空间余弦相似度作为实时李雅普诺夫代理信号，动态识别训练过程中的三个阶段：混沌、过渡与稳定。在各阶段内，采用动量、线性、二次外推等解析预测器尝试预测未来K步的模型参数；仅当预测结果满足保留损失标准时才被采纳。我们在GPT-2 124M和Qwen 2.5-1.5B上基于WikiText-103数据集进行评估，测试了K ∈ {5, 10, 25, 50, 75, 100}的预测深度。动量预测在两种规模下均失败，预测损失高出实际损失100至10,000倍——表明优化器状态外推存在普遍性数值爆炸。有限差分预测器（线性、二次）则成功克服此问题：在124M模型上，稳定阶段于K=5时达到24%严格接受率；1.5B模型在过渡阶段达37%。模型规模影响显著：GPT-2 124M有34%训练时间处于稳定态，而Qwen 1.5B仅有0–2个检查点进入稳定态，64%时间处于混沌态。大模型虽在可预测阶段更易预测，但可预测时机更少，实用瓶颈从预测精度转向可用阶段的分布。跨随机种子结果高度一致（验证损失方差小于1%），三阶段框架在不同种子间相位边界差异不超过±50步。
原文摘要 · Abstract (English)
We introduce Leap+Verify, a framework that applies speculative execution -- predicting future model weights and validating predictions before acceptance -- to accelerate neural network training. Inspired by speculative decoding in language model inference and by the Automatically Scalable Computation (ASC) architecture for program execution, Leap+Verify decomposes training into three dynamically detected regimes (chaotic, transition, stable) using activation-space cosine similarity as a real-time Lyapunov proxy signal. Within each regime, analytic weight predictors (momentum, linear, quadratic extrapolation) attempt to forecast model parameters K training steps ahead; predictions are accepted only when validated against a held-out loss criterion. We evaluate Leap+Verify on GPT-2 124M and Qwen 2.5-1.5B trained on WikiText-103 across five random seeds, sweeping prediction depth K in {5, 10, 25, 50, 75, 100}. Momentum-based prediction (Adam moment extrapolation) fails catastrophically at both scales, with predicted losses exceeding actuals by 100-10,000x -- a universal norm explosion in optimizer-state extrapolation. Finite-difference predictors (linear, quadratic) succeed where momentum fails: at 124M, they achieve 24% strict acceptance at K=5 in stable regimes; at 1.5B, they achieve 37% strict acceptance in transition regimes. The scale-dependent finding is in regime distribution: GPT-2 124M spends 34% of training in stable regime, while Qwen 1.5B spends 64% in chaotic regime and reaches stable in only 0-2 of 40 checkpoints. Larger models are more predictable when predictable, but less often predictable -- the practical bottleneck shifts from predictor accuracy to regime availability. Cross-seed results are highly consistent (less than 1% validation loss variance), and the three-regime framework produces identical phase boundaries (plus or minus 50 steps) across seeds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。