专家数量越多，模型记东西越强，但推理能力到一定程度就不再提升。
Mixture of Parrots: Experts improve memorization more than reasoning
- 通过增加专家数量，在参数不变前提下提升记忆能力
- 推理能力在专家数增多后趋于饱和，无法持续提升
- 适合知识密集型任务，不适合复杂逻辑推理
Mixture-of-Experts（MoE）架构可在几乎不增加计算开销的前提下显著提升模型总参数量。本文发现，在固定活跃参数数量的情况下，随着专家数量增加，模型的记忆性能持续提升，而推理能力则趋于饱和。理论上，某些图问题无论多少宽度的专家都无法解决，但相同任务可由略宽的稠密模型轻松处理。在合成图问题和闭卷检索等记忆密集型任务上，我们验证了这一现象：少量活跃参数搭配大量专家能高效记忆数据。预训练多组MoE与稠密Transformer并在数学和自然语言基准上评估，结果显示增加专家数有助于知识密集型任务，但对推理任务无明显提升。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what performance tradeoffs, if any, exist between MoEs and standard dense transformers. In this paper, we show that as we increase the number of experts (while fixing the number of active parameters), the memorization performance consistently increases while the reasoning capabilities saturate. We begin by analyzing the theoretical limitations of MoEs at reasoning. We prove that there exist graph problems that cannot be solved by any number of experts of a certain width; however, the same task can be easily solved by a dense model with a slightly larger width. On the other hand, we find that on memory-intensive tasks, MoEs can effectively leverage a small number of active parameters with a large number of experts to memorize the data. We empirically validate these findings on synthetic graph problems and memory-intensive closed book retrieval tasks. Lastly, we pre-train a series of MoEs and dense transformers and evaluate them on commonly used benchmarks in math and natural language. We find that increasing the number of experts helps solve knowledge-intensive tasks, but fails to yield the same benefits for reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。