用语义锚点相似度路由，让MoE模型决策可解释且更稳定。
Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
- 通过词元与可学习语义锚点的余弦相似度实现路由，决策过程透明可追踪。
- 在WikiText-103上性能媲美传统线性路由，降低死专家比例至0-6%。
- 适合关注模型可解释性、路由机制稳定性的研究者和工程应用。
Mixture-of-Experts（MoE）模型通过稀疏激活提升效率，但其门控函数难以揭示路由决策依据。本文提出语义共振架构（SRA），通过词元表示与可学习语义锚点之间的余弦相似度路由，使每一步路由均可追溯至相似度分数。我们在WikiText-103上评估了17种配置。在3种子×4配置的受控对比中（256个专家，$D_{ff}=256$），余弦路由与标准线性路由表现相当（$K=1 \to 4$：12.57±0.03 vs 12.45±0.03；$K=2 \to 4$：12.52±0.02 vs 12.57±0.02）。训练方案而非路由函数决定专业化质量，而余弦路由具备天然可解释性。我们引入带通路由损失（约束专家利用率上下限），将死专家比例从30-45%降至0-6%，并适用于两种路由方式。路由空间评估显示，余弦路由在深层中显著提升词级子词一致性（$p < 0.001$），44-54%的专业化为句法而非语义。扩展分析表明，余弦路由具有更稳定的路由器饱和度与更紧致的每专家词汇分布——源于余弦相似度范围有界。推理时的$k$-sweep实验显示，$k=5$比$k=4$带来0.08-0.16的免费困惑度下降。在OpenWebText上的跨数据集验证确认泛化能力：余弦路由达到可比困惑度（44.88 vs 45.44），带通损失消除死专家，专业化模式保持一致。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models improve efficiency through sparse activation, but their learned gating functions provide limited insight into routing decisions. This work introduces the Semantic Resonance Architecture (SRA), which routes tokens to experts via cosine similarity between token representations and learnable semantic anchors, making every routing decision directly traceable to anchor-token similarity scores. We evaluate SRA on WikiText-103 across 17 configurations. In a controlled multi-seed comparison (3 seeds x 4 configurations, 256 experts, $D_{ff}=256$), cosine routing achieves competitive perplexity with standard linear routing ($12.57 \pm 0.03$ vs $12.45 \pm 0.03$ for $K=1 \to 4$; $12.52 \pm 0.02$ vs $12.57 \pm 0.02$ for $K=2 \to 4$). The training recipe -- not the routing function -- drives specialization quality, while cosine routing provides inherent inspectability. We introduce a bandpass routing loss -- a floor-and-ceiling corridor on expert utilization -- that reduces dead experts from 30-45% to 0-6% and transfers to both routing types. Routing-space evaluation shows cosine routing provides significantly better word-level subtoken coherence in deeper layers ($p < 0.001$), with 44-54% of expert specialization being syntactic rather than semantic. Extended analysis reveals cosine routing maintains more stable router saturation and tighter per-expert vocabulary distributions -- structural advantages from the bounded cosine similarity range. An inference-time $k$-sweep shows that $k=5$ yields a free 0.08-0.16 perplexity gain over $k=4$. Cross-dataset validation on OpenWebText confirms generalization: cosine routing achieves comparable perplexity (44.88 vs 45.44), the bandpass loss eliminates dead experts, and specialization patterns are preserved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。