对比稀疏MoE与稠密模型内部表示，发现前者更专注、特征更少但激活更密集。
Sparse Crosscoders for diffing MoEs and Dense models
- 用跨空间稀疏自编码器联合建模多层激活空间
- MoE仅学出约13%的独有特征，且激活密度更高
- 适合研究模型内部表征差异或高效架构设计
混合专家（MoE）通过稀疏专家路由实现参数高效扩展，但其内部表示机制仍远不如稠密模型清晰。本文使用一种变体稀疏自编码器——交叉编码器（crosscoders），系统比较了MoE与稠密模型的内部结构。在代码、科学文本和英文故事共10亿词元的数据上，训练了5层等效活跃参数的稠密模型与MoE。采用带显式共享特征的BatchTopK交叉编码器，实现了约87%的方差解释率，并揭示出显著的特征组织差异：MoE学习到的独有特征明显少于稠密模型；而MoE特有特征的激活密度高于共享特征，稠密模型特有特征则更低。分析表明，MoE发展出更专业化、聚焦的表示，而稠密模型则将信息分布于更广泛、通用的特征中。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) achieve parameter-efficient scaling through sparse expert routing, yet their internal representations remain poorly understood compared to dense models. We present a systematic comparison of MoE and dense model internals using crosscoders, a variant of sparse autoencoders, that jointly models multiple activation spaces. We train 5-layer dense and MoEs (equal active parameters) on 1B tokens across code, scientific text, and english stories. Using BatchTopK crosscoders with explicitly designated shared features, we achieve $\sim 87\%$ fractional variance explained and uncover concrete differences in feature organization. The MoE learns significantly fewer unique features compared to the dense model. MoE-specific features also exhibit higher activation density than shared features, whereas dense-specific features show lower density. Our analysis reveals that MoEs develop more specialized, focused representations while dense models distribute information across broader, more general-purpose features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。