用线性时间构建大模型变体，无需微调即可实现高效推理。
Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors
- 通过独立插值权重，动态调节父模型语义特征。
- 生成的子模型几乎全部可用，且推理速度提升40%。
- 适合需要快速部署和优化推理效率的开发者。
预训练大型语言模型时，计算单个8比特权重需消耗10¹³至10¹⁵次浮点运算，成本极高。为更好利用已有预训练模型，我们提出「专家集合」（Assembly-of-Experts, AoE）构造方法，可在线性时间内生成具备新能力的子模型。该方法对模型权重张量进行独立插值，可增强或抑制父模型的语义特征。调整权重来源比例后，部分行为属性呈渐变变化，而某些特性则出现突变式涌现。令人惊讶的是，几乎所有生成的子模型均功能正常且表现良好，使模型空间搜索变得简单直接。我们基于DeepSeek V3-0324与R1模型变体，构建了671B参数的开放权重混合模型Chimera。该子模型仅继承R1的路由专家张量，但仍达到接近R1的智能水平，同时输出令牌数减少约40%，接近V3的推理速度。该模型未经过任何微调或蒸馏，却展现出异常紧凑有序的推理过程。
原文摘要 · Abstract (English)
Requiring $10^{13}$-$10^{15}$ FLOPs to calculate one 8 bit weight in an LLM during pretraining is extremely expensive and seems inefficient. To better leverage the huge investments made into pretrained models, we develop the new "Assembly-of-Experts" (AoE) construction method to create capable child variants of existing Mixture-of-Experts parent models in linear time. Model weight tensors get interpolated individually, allowing to enhance or suppress semantic features of the parents. Varying the proportion of weights taken from the parent models, we observe some properties of the AoE child model changing gradually, while other behavioral traits emerge with a sharp transition. Surprisingly, nearly every generated model is functional and capable, which makes searching the model space straightforward. We construct the DeepSeek R1T "Chimera", a 671B open-weights hybrid model combining DeepSeek's V3-0324 and R1 model variants. The child inherits only the routed expert tensors of R1, but still achieves about R1-level intelligence. At the same time, it uses about 40\% fewer output tokens, close to V3 speed. Constructed without any fine-tuning or distillation, the Chimera exhibits surprisingly compact, orderly reasoning compared to its parent models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。