arXiv:2605.02124cs.LGcs.AI2026-05被引 1

揭示稀疏专家模型中软路由转硬路由的几何机制,指出边界层影响远大于温度本身。

Soft-to-Hard Routing in Sparse Mixture-of-Experts Models

  • 基于边界层微分几何分析软硬路由转换过程
  • 零温近似误差由路由界面邻域概率决定,非仅温度影响
  • 适用于研究专家混合模型优化与稳定性问题的研究者

当温度趋近于零时,Softmax路由趋向于硬性top-1路由,但在路由器平局处存在奇异极限。本文为群体平方损失专家混合回归中的这一软到硬极限发展了边界层微积分。对于具有logits $a_k(x;ϕ)$的路由器,关键局部量是前两名的边际差 $Δ(x;ϕ)$,关键全局量是边界质量 $\mathbb{P}(Δ(X;ϕ)\le w)$。在光滑性和横截性假设下,利用共面积和管状邻域估计,揭示该质量随带宽的变化规律;在二元情况下,主导系数为路由界面的显式曲面积分。这些几何估计给出了软目标 $L_τ$ 与硬目标 $L_0$ 的定量比较,包括在边际尾部条件下 $O(τ^α)$ 的统一界,并证明了在紧致参数空间上软目标的 $Γ$-收敛。主要结论是:零温近似受 $O(τ)$ 邻域内路由界面所携带的概率控制,而非温度本身。分离出这一边界层部分后,我们给出从硬路由到小温度软路由的条件景观转移定理,以及一个简化双专家高斯计算,说明局部对称性破缺现象。合成诊断仅用作对边界层预测的可控验证。

原文摘要 · Abstract (English)

Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus for this soft-to-hard limit in population squared-loss mixture-of-experts regression. For a router with logits $a_k(x;ϕ)$, the relevant local quantity is the top-two margin $Δ(x;ϕ)$, and the relevant global quantity is the boundary mass $\mathbb{P}(Δ(X;ϕ)\le w)$. Under smoothness and transversality assumptions, coarea and tubular-neighborhood estimates show how this mass scales with the slab width; in the binary case the leading coefficient is an explicit surface integral over the routing interface. These geometric estimates give quantitative bounds between the soft objective $L_τ$ and the hard objective $L_0$, including an $O(τ^α)$ uniform comparison under a margin-tail condition, and yield $Γ$-convergence of the soft objectives on compact parameter spaces. The main conclusion is that the zero-temperature approximation is controlled by the probability carried by an $O(τ)$ neighborhood of the routing interfaces, not by temperature alone. After isolating this boundary-layer part of the problem, we record a conditional landscape-transfer theorem from hard to small-temperature soft routing and a reduced two-expert Gaussian calculation illustrating local symmetry breaking. Synthetic diagnostics are included only as controlled checks of the boundary-layer predictions.

专家混合路由机制边界层分析优化理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。