用自蒸馏让搜索增强模型自我进化,无需复杂外部机制
Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning
- 通过自蒸馏在策略训练中注入密集监督信号
- 在7个问答基准上达0.440平均EM,超越所有开源基线
- 方法极简,适合资源有限或追求轻量训练的研究者
后训练已成为将语言模型转化为高效搜索增强推理代理的主流方法。近期研究通过引入外部监督、附加辅助模块、重构推理路径或手工设计奖励等复杂机制进一步提升性能。这些改进虽有效,但显著增加训练开销并依赖特定资源。本文提出Search-E1,一种仅通过标准GRPO与在线策略自蒸馏(OPSD)交替进行的自进化方法。每轮GRPO后,策略在自身训练问题上自主推理,利用令牌级前向KL散度将其推断分布对齐到更高效兄弟轨迹的特权上下文分布。该过程自然产生密集的每步监督信号。在7个QA基准上,Search-E1使用Qwen2.5-3B模型达到0.440平均精确匹配(EM),超越所有同规模开源基线。代码与完整版本即将公开。
原文摘要 · Abstract (English)
Post-training has become the dominant recipe for turning a language model into a competent search-augmented reasoning agent. A line of recent work pushes its performance further by adding elaborate machinery on top of this standard pipeline. These augmentations import external supervision from stronger external systems, attach auxiliary modules such as process reward models or retrospective critics, restructure the rollout itself with tree search or multi-stage curricula, or shape the reward with hand-crafted bonuses and penalties. Each addition delivers a measurable gain, but each also inflates the training pipeline and ties the recipe to resources or designs that may not always be available. We take a step back and ask whether any of this machinery is actually necessary, and propose Search-E1, a self-evolution method that lets a search-augmented agent improve through only vanilla GRPO interleaved with on-policy self-distillation (OPSD). After each GRPO round, the policy rolls out on its own training questions. A token-level forward KL objective then aligns the policy's inference-time distribution to its own distribution under a privileged context that exposes a more efficient sibling trajectory. Despite this simplicity, the procedure naturally provides dense per-step supervision. On seven QA benchmarks, Search-E1 reaches 0.440 average EM with Qwen2.5-3B, surpassing all open-source baselines at both scales. Code and complete version will be made public soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。