通过文本提示模拟路由机制，发现模型内部激活更密集而非稀疏，稳定性关联弱。
Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States
- 用文本提示模拟路由信号，探测模型内部状态变化
- 早期到中期层激活密度上升，但输出稳定性未提升
- 适合研究模型路由设计与不确定性估计的开发者
路由广泛用于扩展大语言模型，如专家混合门控和多模型/工具选择。普遍认为路由至任务专家会引发更稀疏的内部计算，从而产生更确定、稳定的输出（稀疏性-确定性假说）。我们通过在冻结的指令微调大模型前注入路由风格的元提示，作为路由信号的文本代理来检验该假说。量化了（C1）内部密度（通过激活稀疏度）、（C2）领域关键词注意力，以及（C3）输出稳定性（通过预测熵和语义变异）。在包含三个指令微调模型（Qwen3-8B、Llama-3.1-8B-Instruct、Mistral-7B-Instruct-v0.2）的RouterEval子集上，元提示一致地使早期/中层表示更密集，而非更稀疏；自然语言专家指令通常比结构化标签更强。注意力响应呈现异质性：Qwen/Llama降低关键词注意力，而Mistral增强它。最终，密集化与稳定性的关联较弱，仅在Qwen中出现，而Llama和Mistral的相关系数接近零。我们提出RIDE作为校准路由设计与不确定性估计的诊断探针。
原文摘要 · Abstract (English)
Routing is widely used to scale large language models, from Mixture-of-Experts gating to multi-model/tool selection. A common belief is that routing to a task ``expert'' activates sparser internal computation and thus yields more certain and stable outputs (the Sparsity--Certainty Hypothesis). We test this belief by injecting routing-style meta prompts as a textual proxy for routing signals in front of frozen instruction-tuned LLMs. We quantify (C1) internal density via activation sparsity, (C2) domain-keyword attention, and (C3) output stability via predictive entropy and semantic variation. On a RouterEval subset with three instruction-tuned models (Qwen3-8B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.2), meta prompts consistently densify early/middle-layer representations rather than increasing sparsity; natural-language expert instructions are often stronger than structured tags. Attention responses are heterogeneous: Qwen/Llama reduce keyword attention, while Mistral reinforces it. Finally, the densification--stability link is weak and appears only in Qwen, with near-zero correlations in Llama and Mistral. We present RIDE as a diagnostic probe for calibrating routing design and uncertainty estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。