动态调整每条输入的专家数量，降低推理延迟而不损失精度。
Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
- 根据路由概率分布自动识别关键拐点，动态决定激活专家数。
- 在六个基准上平均延迟降低5.3%，精度保持不变。
- 无需训练，可直接部署于现有MoE模型，适合高效推理场景。
Mixture-of-Experts (MoE) 模型通过仅激活部分专家实现模型扩展，同时保持较低的推理计算量。然而，传统路由依赖固定的 top-k 选择，导致模型在无论相关专家多少的情况下都消耗相同计算量。本文提出一种无需训练的推理时改进方法——肘部路由（elbow-based routing），能够根据每个输入词元动态调整激活的专家数量。该方法分析排序后的路由概率分布，识别出区分高、低概率专家的拐点。我们发现大多数路由分布均存在明显的拐点，适合此策略，并从理论上和实证上证明其能保持专家负载均衡。在一款先进 MoE 模型上的实验表明，该方法在六个基准测试中实现了平均 5.3% 的延迟降低，同时维持了原有精度。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models enable model scaling while maintaining low inference-time compute by activating only a subset of experts per token. However, conventional routing relies on a fixed top-k selection, forcing the model to spend the same compute regardless of how many experts are relevant. We introduce elbow-based routing, a training-free inference-time modification that dynamically adjusts the number of experts on a per-token basis. Our method examines the sorted router probability distribution and identifies an elbow point that separates high- and low-probability experts. We find that most router distributions exhibit clear inflection points suitable for this strategy, and we show both theoretically and empirically that elbow-based routing preserves expert load balance. Experiments on a state-of-the-art MoE model demonstrate an average latency reduction of 5.3% while maintaining accuracy across six benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。