让小模型也能像智能体一样搜索，靠的是教师指导的强化学习优化。
Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities
- 用大模型示范初始化+持续引导，提升小模型初始表现。
- 小模型经训练后能实现复杂搜索行为，部分任务超越原大模型。
- 适合资源受限场景下的智能体式问答系统开发。
强化学习已成为激发语言模型具备搜索与规划等智能体式检索增强生成（RAG）行为的主要后训练方法。尽管在大模型上取得成功，将其应用于紧凑模型（如0.5–1B参数）仍面临挑战：初始性能差导致奖励稀疏、训练不稳定。为此，本文提出蒸馏引导的策略优化（DGPO），通过教师示范进行冷启动初始化，并在策略优化过程中持续提供教师指导。为分析紧凑模型如何保留智能体能力，我们引入细粒度评估指标Agentic RAG Capabilities（ARC），用于分析推理、搜索协调与响应生成。大量实验表明，DGPO使紧凑模型实现复杂智能体式搜索行为，甚至在某些任务中优于更大教师模型。该方法使资源受限环境下的智能体式RAG成为可能。
原文摘要 · Abstract (English)
Reinforcement Learning has emerged as a dominant post-training approach to elicit agentic RAG behaviors such as search and planning from language models. Despite its success with larger models, applying RL to compact models (e.g., 0.5--1B parameters) presents unique challenges. The compact models exhibit poor initial performance, resulting in sparse rewards and unstable training. To overcome these difficulties, we propose Distillation-Guided Policy Optimization (DGPO), which employs cold-start initialization from teacher demonstrations and continuous teacher guidance during policy optimization. To understand how compact models preserve agentic behavior, we introduce Agentic RAG Capabilities (ARC), a fine-grained metric analyzing reasoning, search coordination, and response synthesis. Comprehensive experiments demonstrate that DGPO enables compact models to achieve sophisticated agentic search behaviors, even outperforming the larger teacher model in some cases. DGPO makes agentic RAG feasible in computing resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。