大模型转小模型，初始化比权重迁移更关键。
Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't
- 用线性映射转换大模型权重，但会破坏注意力等结构
- 3000万tokens下仍优于从头训练，节省大量计算
- 适合低预算场景，尤其在小模型规模时优势明显
模型家族通常按大小逐个从头训练。能否将预训练的大模型直接转换为较小版本？我们完整分析了Pythia中1.4B→410M的转换过程。不同规模间表征对齐度高（岭回归R²=0.84），但参数对齐弱。密集权重投影功能上具有破坏性，且比特精确控制显示这并非合成误差：基底混叠会破坏旋转位置编码、每头结构、GELU和层归一化。经最优线性算子处理后，权重残差在随机置换控制下与噪声无异。因此，转换价值存在于初始化中。在相同预算下继续预训练，我们解耦出两个独立杠杆：最小二乘补偿（功能杠杆，零样本最佳）与方差保持缩放（动力学杠杆，终点表现最佳）。补偿在低预算下高效，3000万令牌时，在宽度缩减对（84.0±1.8 vs. 89.7±3.7，3/3种子）和深度缩减对（109.3 vs. 117.9，3/3种子）均胜过最强子克隆变体，以更少令牌达到同等质量。33倍更大预算下两者趋同（40.0 vs. 40.0），远超从头训练，低预算下可领先18倍，随规模增大差距缩小。我们还划定方法边界：约5倍于源模型规模（6.9B→1.4B）时叠加双杠杆会过度修正，与补偿求解在大宽度下的病态性一致，提示需引入维度感知正则化。在相同预算下，该初始化方案优于结构化剪枝+蒸馏标准流程，并进一步提升性能。代码、检查点及冻结评估语料均已公开。
原文摘要 · Abstract (English)
Model families are typically trained size by size, each from scratch. Can a pretrained large model instead be converted into a smaller sibling? We characterize the 1.4B->410M conversion in Pythia end to end. Representations align strongly across sizes (ridge R^2=0.84) while parameters align weakly. Dense weight projection is functionally destructive, and a bit-exact control shows this is not an assembly artifact: basis mixing breaks rotary, per-head, GELU, and LayerNorm structure. After the best-fit linear operator, weight residuals are statistically indistinguishable from noise under shuffle controls. Conversion value therefore lives in initialization. In matched-budget continued pre-training we decompose conversion into two independent levers: least-squares compensation (function lever, best zero-shot) and variance-preserving rescale (dynamics lever, best endpoints). Compensation is a token-efficient, low-budget win rather than a universal one. At 30M tokens it beats the strongest subcloning variant on both a width-reduced pair (84.0 +/- 1.8 vs. 89.7 +/- 3.7, 3/3 seeds) and a held-out depth-reduced pair (109.3 vs. 117.9, 3/3 seeds), reaching a given quality with fewer tokens. At a 33x larger budget the two converge to parity (40.0 vs. 40.0), both far ahead of from-scratch, which transfer initialization always beats: by up to 18x at low budget, with the margin narrowing at convergence and at the largest scale. We also map the method's boundary. At about 5x the donor scale (6.9B->1.4B) stacking both levers over-corrects, consistent with ill-conditioning of the compensation solve at large width, which points to dimension-aware regularization as a fix. At matched budget our initialization also beats structured pruning with distillation, the standard pipeline, and improves further combined with it. Code, checkpoints, and the frozen evaluation corpus are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。