用多智能体协议蒸馏,让开源模型学会大厂闭源模型的推理能力。
From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search

- 设计结构化协议作为中间表示,实现跨模型推理能力迁移。
- 在7个问答数据集上,4B模型达44.4%准确率,优于主流方法。
- 适合想提升开源模型推理性能的研究者与开发者使用。
智能体搜索通过融合多步推理与检索,使大语言模型能解决知识密集型任务,但基于结果的强化学习仅提供稀疏监督。知识蒸馏可提供更密集的指导,而具备强大推理能力的闭源模型是理想的教师。然而,传统基于对数概率匹配的方法受限于隐藏对数和不匹配的分词器,而直接模仿自然语言轨迹则仅传递表面风格而非核心推理能力。为解决异构蒸馏问题并弥合分布差距,我们提出多智能体协议蒸馏(MAPD),一种联合蒸馏与强化学习的框架,采用结构化、风格归一化的协议作为中间表示。离线多智能体系统(MAS)分解查询、检索证据、修复失败搜索,并将探索过程转化为包含任务类型、推理计划和提取性事实的JSON协议。训练时,仅将协议提供给学生策略的特权分支,其词分布为强化学习目标提供密集蒸馏信号。在七个问答基准上的大量评估表明,MAPD持续优于对比方法,在Qwen3-1.7B和Qwen3-4B上分别达到39.4%和44.4%的平均成功率。关键的是,该框架在多种闭源教师间具有强泛化能力,有效缓解学生策略的风格漂移与冗长退化问题。
原文摘要 · Abstract (English)
Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of 39.4\% on Qwen3-1.7B and 44.4\% on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。