MoE模型的专家选择能泄露近9成原文，比预想严重得多。
Expert Selections In MoE Models Reveal (Almost) As Much As Text
- 用专家路由决策重建文本，不依赖原始输入。
- 用Transformer解码器在32词序列上复原率达91.2%（前10名94.8%）。
- 适用于隐私敏感场景，如分布式推理中的侧信道攻击。
我们提出一种针对混合专家（MoE）语言模型的文本重构攻击，仅凭专家选择即可恢复令牌。在MoE模型中，每个令牌被路由至一组专家子网络；我们发现这些路由决策泄露的信息远超以往认知。先前研究使用逻辑回归仅实现有限重构；我们表明，三层MLP可将准确率提升至63.1%顶1准确率，而基于Transformer的序列解码器在训练1亿令牌后，对OpenWebText的32词序列可实现91.2%顶1准确率（顶10为94.8%）。该结果将MoE路由与嵌入反演研究相连接。我们列举了实际泄露场景（如分布式推理和侧信道），并证明添加噪声虽能降低但无法消除重构能力。结果表明，部署中的专家选择应视同原文一样敏感。
原文摘要 · Abstract (English)
We present a text-reconstruction attack on mixture-of-experts (MoE) language models that recovers tokens from expert selections alone. In MoE models, each token is routed to a subset of expert subnetworks; we show these routing decisions leak substantially more information than previously understood. Prior work using logistic regression achieves limited reconstruction; we show that a 3-layer MLP improves this to 63.1% top-1 accuracy, and that a transformer-based sequence decoder recovers 91.2% of tokens top-1 (94.8% top-10) on 32-token sequences from OpenWebText after training on 100M tokens. These results connect MoE routing to the broader literature on embedding inversion. We outline practical leakage scenarios (e.g., distributed inference and side channels) and show that adding noise reduces but does not eliminate reconstruction. Our findings suggest that expert selections in MoE deployments should be treated as sensitive as the underlying text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。