不同专家合并顺序导致模型输出差异，揭示了稀疏MoE的数值稳定性关键因素。
From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
- 通过固定局部状态，改变聚合语义，分离操作数表示与累加精度
- 同一提示下，不同模式产生202种离职、113种招聘等多样后续路径
- 首次实现跨标记延续状态的精确重建，验证了数值兼容性机制
数学等价的专家缩减顺序可导致稀疏MoE执行结果明显不同。本研究在原生DeepSeek-V4-Flash中冻结局部MoE状态，仅改变聚合语义，提出四种方案分离操作数表示与累加器精度。在第5层分叉处，720个A模式顺序产生10个延续基底；720个B模式顺序形成360个精确结构类别和11个基底。针对一个中文提示，B类分裂为202种裁员、113种招聘及45种其他延续。最大L-infinity B分支选择可将50个提示按8、16、32个标记区分。每种方案下192条持久轨迹中，P32、A、B模式均改变原始参考路径，而C模式保持路径、标记序列与文本不变。对C模式的192条轨迹检查显示，其与原生MoE、post-mHC、next-router及LM状态逐位匹配。对某一受控B分支，精确恢复post-mHC端点可复现下游轨迹。在下一解码边界，精确重构该分支全持久状态后，给定自然生成输入，在七步内实现301个post-mHC状态、301个持久状态检查点、301条路径、预测结果与文本完全一致。这些控制实验表明post-mHC为标记内边界，全持久状态为跨标记延续边界。相同标记未必对应相同自回归状态：分歧可穿越标记边界并在后续显现。结果表明，专家操作数转换、累加精度与缩减顺序构成稀疏MoE运行时与硬件后端的数值兼容性契约。研究建立可控因果可能性，非部署实例；C模式的顺序无关性限于评估的六项状态与调度。
原文摘要 · Abstract (English)
Mathematically equivalent expert-reduction orders can produce observably different sparse-MoE executions. We isolate this effect in native DeepSeek-V4-Flash by freezing local MoE state and varying only aggregation semantics. Four schemes separate operand representation from accumulator precision. At one layer-5 fork, 720 A-mode orders yield 10 continuation basins; 720 B-mode orders form 360 exact structural classes and 11 basins. Under one Chinese prompt, the B classes split into 202 layoffs, 113 hiring, and 45 other continuations. Maximum-L-infinity B-branch selection separates 12, 24, and 36 of 50 prompts by 8, 16, and 32 tokens. Across 192 persistent trajectories per scheme, P32, A, and B change every native-reference route trajectory, while C preserves routes, token sequences, and texts. A separate 192-trajectory C check matches native MoE, post-mHC, next-router, and LM states bitwise. For one controlled B branch, exact post-mHC endpoint reconstruction reproduces the measured downstream trajectory. At the next decode boundary, exact FP64 reconstruction of the branch's full persistent state yields agreement for 301 downstream post-mHC states, 301 persistent-state checkpoints, 301 routes, predictions, and text over seven steps, given the same naturally generated next input. These controls identify post-mHC as an intra-token boundary and full persistent state as a cross-token continuation boundary. Identical tokens need not imply identical autoregressive state: divergence can survive a token boundary and become visible later. These results make expert operand conversion, accumulator precision, and reduction order part of a numerical compatibility contract for sparse-MoE runtimes and hardware backends. They establish controlled causal possibility, not deployment incidence; C's order invariance is limited to evaluated six-term states and schedules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。