发现小模型量化训练中学习率调度可通用，不需针对精度单独调
Mapping the Schedule x Bit-Width Boundary in Sub-100M Quantisation-Aware Training
- 在不同精度和规模下测试学习率调度，发现33%退火比例始终最优
- 50M以上模型用INT4时，33%退火显著优于其他方案（10/10种子）
- 50M以下模型调度选择无显著差异，结果受随机噪声主导
我们检验了在从初始化开始的量化感知训练（QAT）中，学习率调度是否依赖于位宽，针对参数量小于100M的解码器语言模型。第二阶段的720次因子网格实验（覆盖位宽×退火比例×学习率幅度×模型规模×种子，包括FP16/INT8/INT6、15M-100M、5个种子）发现，每个（位宽，规模）组合下的最优退火比例均为33%。最初假设——INT6 QAT需要不同于高精度训练的调度——在FP16/INT8/INT6上被证伪。第五阶段625次跟进实验在五个维度上验证该零效应：优化器（AdamW）、调度形状（余弦）、训练长度（最多延长9倍）、扩展规模范围（5M-350M），以及从3M到100M的INT4实验。零效应在所有三种设置变化下均保持稳健。INT6的性能损失符合对数线性缩放律，其在第二阶段的拟合能准确预测第五阶段5个未见规模（5M、8M、175M、250M、350M）的损失值，且均在95%预测区间内（5/5）。对于INT4，情况更清晰：在50M和100M时，33%退火显著最优（配对z统计量~12-15，10/10种子）；低于50M时，在3M至30M共六个规模上，单个规模均无统计显著的调度偏好，各规模平均损失波动在种子级噪声范围内。因此，边界是50M以下的噪声主导区与50M及以上明确的wd33最优区之间的过渡，而非清晰的wd10区域。权重到网格距离探测否定了最简单机制（快速网格捕获）——预退火阶段，INT6-QAT权重与INT6网格的距离与FP16权重几乎相同（比值~1.04）。实际建议：在小于100M的规模下，仅在FP16上调试一次学习率调度，即可直接用于INT8/INT6 QAT；对于50M以上的INT4使用wd33；低于50M的INT4调度选择无影响。
原文摘要 · Abstract (English)
We test whether the optimal learning-rate schedule depends on bit-width during from-initialisation quantisation-aware training (QAT) for sub-100M decoder language models. A 720-run factorial grid (Phase 2) over bit-width x warmdown fraction x LR magnitude x model size x seed (FP16/INT8/INT6, 15M-100M, 5 seeds) finds the optimal warmdown is 33% at every (bit-width, size) cell. The primary hypothesis -- that INT6 QAT requires a different schedule than higher-precision training -- is falsified at FP16/INT8/INT6. A 625-run follow-up (Phase 5) probes the null along five axes: optimiser (AdamW), schedule shape (cosine), training length (up to 9x more iterations), an extended size sweep (5M-350M), and an INT4 sweep from 3M to 100M. The null is robust under all three setup changes. The INT6 penalty follows a log-linear scaling law whose fit on Phase 2 predicts the five held-out Phase 5 sizes (5M, 8M, 175M, 250M, 350M) within their 95% prediction intervals (5/5). For INT4 the picture is sharper than the higher precisions: at 50M and 100M, wd33 is decisively optimal (paired z ~ 12-15, 10/10 seeds); below 50M, across the six tested sizes from 3M to 30M, no individual size shows a statistically significant schedule preference and the per-size mean penalty oscillates within seed-level noise. The boundary is therefore a transition between a noise-dominated regime below 50M and a decisive wd33 regime at and above 50M, not a clean wd10 region. A weight-to-grid-distance probe falsifies the simplest mechanism for the FP16/INT8/INT6 null result (rapid grid-snapping): pre-warmdown, INT6-QAT weights sit at essentially the same distance from the INT6 grid as FP16 weights (ratio ~ 1.04). Practical recommendation: at sub-100M scale, tune the LR schedule once at FP16 and apply unchanged to INT8/INT6 QAT; for INT4 at 50M+ use wd33; for INT4 below 50M the schedule choice is in the noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。