提出可预测专家模型融合效果的定量模型，帮开发者提前评估合作价值。
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
- 基于分歧度构建收益公式，预估融合后性能提升
- 多专家融合平均提升7.7%，跨语言融合最高增21.8%
- 支持轻量路由，适合分布式协作训练场景
独立训练的领域专家模型可在后期融合为单一模型，性能超越任一单个专家，且增益可预测：增益 = 0.82 × 分歧度 - 2.72（R² = 0.856，n=6，分歧度3-26%）。当分歧度低于3.3%时，增益趋近于零。KALAVAI协议中，贡献者独立微调共享检查点副本，提交后进行轻量级MoE路由（500步）。增益稳定：410M模型+7.72%（±0.02%，3种子），1B模型+7.49%（±0.01%，3种子），6.9B模型+6.53%，均优于最优专家。路由器匹配领域-真值路由误差小于10⁻⁵纳特。跨语言融合（泰米尔语/约鲁巴语/威尔士语/代码）达+21.76%，约鲁巴语困惑度从41.9降至7.7。20人联邦实现+16.71%（±0.07pp，3种子）。协议有三要求：共享初始化必要，检查点不匹配会降低路由性能；冻结层在1万步内可选，超过则有益；学习型路由必需：均匀平均比最优专家低-1.2%，而任何训练过的路由器均能达到真值最优分配。
原文摘要 · Abstract (English)
Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist, and the gain is predictable: gain = 0.82 x divergence - 2.72 (R^2 = 0.856, n=6, 3-26% divergence). This enables practitioners to estimate cooperative value before committing compute. Below ~3.3% divergence, gains approach zero.In the KALAVAI protocol, contributors fine-tune copies of a shared checkpoint independently, then submit for lightweight MoE routing (500 steps). Gains are consistent: +7.72% at 410M (+/-0.02%, 3 seeds), +7.49% at 1B (+/-0.01%, 3 seeds), +6.53% at 6.9B, each over the best specialist. The router matches domain-oracle routing within <10^{-5} nats. Cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves +21.76%, with Yoruba perplexity falling 41.9 to 7.7. A 20-contributor federation achieves +16.71% (+/-0.07pp, 3 seeds).Three requirements bound the protocol. Shared initialisation is necessary: checkpoint mismatch degrades routing. Frozen layers are optional below ~10,000 steps and beneficial beyond. Learned routing is essential: uniform averaging degrades by -1.2% vs. best specialist, while any trained router achieves oracle-optimal assignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。