让多个语言模型异步训练，推理时只用部分参数，效果更好。
No Need to Talk: Asynchronous Mixture of Language Models
- 各模型专注不同数据，无需高速通信，实现近乎异步训练。
- 相同计算量下，困惑度更低，75%任务表现优于基线模型。
- 无需聚类或元数据，适合真实场景部署，推理成本几乎不变。
我们提出SMALLTALK LM，一种近乎异步的混合语言模型训练方法。每个模型专精于数据分布的不同部分，无需节点间高带宽通信。推理时，轻量级路由根据短前缀将序列分配给单个专家，自然仅使用整体模型的一小部分参数。与以往异步LLM训练方法不同，我们的路由机制不依赖全语料库聚类或元数据，更适用于真实应用。在语言建模实验中，SMALLTALK LM 在相同总训练浮点运算量（FLOPs）下，显著降低困惑度，且推理成本几乎相同。下游评估中，在75%的任务上优于密集基线模型。
原文摘要 · Abstract (English)
We introduce SMALLTALK LM, an innovative method for training a mixture of language models in an almost asynchronous manner. Each model of the mixture specializes in distinct parts of the data distribution, without the need for high-bandwidth communication between the nodes training each model. At inference, a lightweight router directs a given sequence to a single expert, according to a short prefix. This inference scheme naturally uses a fraction of the parameters from the overall mixture model. Unlike prior works on asynchronous LLM training, our routing method does not rely on full corpus clustering or access to metadata, making it more suitable for real-world applications. Our experiments on language modeling demonstrate that SMALLTALK LM achieves significantly lower perplexity than dense model baselines for the same total training FLOPs and an almost identical inference cost. Finally, in our downstream evaluations we outperform the dense baseline on 75% of the tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。