揭示多专家协作中谁真正关键,而非仅看谁被频繁调用。
Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration
- 将专家协作视为可分析的计算过程,分离出交互结构与功能归属。
- 发现常被调用的专家未必关键,少数稀疏调用者反而影响全局结构。
- 适合研究模型协作机制、提升系统可解释性的研究人员。
多专家系统通过多个大语言模型协同解决复杂任务,但其专家间交互与调度策略仍不透明。本文提出INFORM可解释性分析方法,将调度过程视为显式可分析的计算,实现对专家交互结构、执行顺序与功能归属的解耦。在GSM8K、HumanEval和MMLU上,使用10个来自LLaMA-3.1 8B、Qwen3 8B和DeepSeek-R1 8B的指令微调专家组成的同质联盟,以及涵盖1B-7B参数规模的异质联盟,通过控制解码温度变化进行评估。结果表明,路由主导性无法有效反映功能必要性。关系重要性(由路由质量与交互拓扑捕获)与内在重要性(通过梯度敏感性衡量)存在显著差异:频繁被选中的专家常为交互枢纽但影响有限,而稀疏路由的专家可能在结构上至关重要。调度行为呈异步演化,专家集中先于稳定路由信心出现,且专家排序具有非确定性。针对性消融实验显示,屏蔽内在重要专家会引发远超频繁同伴的交互结构崩溃,证明INFORM能揭示超越准确率的功能与结构依赖。代码已开源。
原文摘要 · Abstract (English)
Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation. However, the orchestration policies governing expert interaction and sequencing remain largely opaque. We introduce INFORM, an interpretability analysis that treats orchestration as an explicit, analyzable computation, enabling the decoupling of expert interaction structure, execution order, and functional attribution. We use INFORM to evaluate an orchestrator on GSM8K, HumanEval, and MMLU using a homogeneous consortium of ten instruction-tuned experts drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B, with controlled decoding-temperature variation, and a secondary heterogeneous consortium spanning 1B-7B parameter models. Across tasks, routing dominance is a poor proxy for functional necessity. We reveal a divergence between relational importance, captured by routing mass and interaction topology, and intrinsic importance, measured via gradient sensitivity: frequently selected experts often act as interaction hubs with limited influence, while sparsely routed experts can be structurally critical. Orchestration behaviors emerge asynchronously, with expert centralization preceding stable routing confidence and expert ordering remaining non-deterministic. Targeted ablations show that masking intrinsically important experts induces disproportionate collapse in interaction structure compared to masking frequent peers, confirming that INFORM exposes functional and structural dependencies beyond accuracy metrics alone. Our code is available at https://github.com/parmanu-lcs2/inform.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。