MixtureKit让任意模型轻松变专家混合，支持三种路由方式。
MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models
- 提供三种专家混合方法：传统路由、细粒度分支路由、完整专家缝合。
- 在阿拉伯-拉丁语混用数据上，BTX模型性能超越基线密集模型。
- 支持自动配置修改与可视化分析，适合研究和开发专家系统。
我们提出MixtureKit，一个模块化开源框架，用于从任意预训练或微调模型构建、训练和分析混合专家（MoE）模型。当前支持三种互补方法：(i) 传统MoE，每Transformer块使用单一路由器选择专家；(ii) BTX（分支-训练-混合），为指定子层引入独立路由器，实现细粒度令牌路由；(iii) BTS（分支-训练-缝合），保持专家完整并引入可训练缝合层，实现枢纽与专家间受控信息交换。MixtureKit自动修改模型配置，修补解码器与因果语言模型类，并保存统一检查点，可直接用于推理或微调。我们还提供可视化界面，用于查看每令牌路由决策、专家权重分布及逐层贡献。在多语言代码切换数据（如阿拉伯-拉丁语）上的实验表明，基于BTX的模型经MixtureKit训练后，在多个基准测试中优于基线密集模型。我们公开发布MixtureKit，作为跨领域MoE系统研究与开发的实用基础。
原文摘要 · Abstract (English)
We introduce MixtureKit, a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned models. MixtureKit currently supports three complementary methods: (i) \emph{Traditional MoE}, which uses a single router per transformer block to select experts, (ii) \emph{BTX} (Branch-Train-Mix), which introduces separate routers for each specified sub-layer enabling fine-grained token routing, and (iii) \emph{BTS} (Branch-Train-Stitch), which keeps experts fully intact and introduces trainable stitch layers for controlled information exchange between hub and experts. MixtureKit automatically modifies the model configuration, patches decoder and causal LM classes, and saves a unified checkpoint ready for inference or fine-tuning. We further provide a visualization interface to inspect per-token routing decisions, expert weight distributions, and layer-wise contributions. Experiments with multilingual code-switched data (e.g. Arabic-Latin) show that a BTX-based model trained using MixtureKit can outperform baseline dense models on multiple benchmarks. We release MixtureKit as a practical foundation for research and development of MoE-based systems across diverse domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。