用合成任务和模拟环境训练小模型，让其具备强智能体能力。
Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards
- 用强模型生成多样且模糊的任务，逼小模型主动追问用户。
- 在模拟环境中训练，小模型在14个任务上超越更大基线模型。
- 适合想低成本训练高效智能体的研究者和开发者。
小型语言模型常难以达到大型昂贵模型的智能体能力。尽管强化学习可提升表现，但受限于两大结构性瓶颈：开源智能体训练数据任务类型单一且易解；真实API缺乏多样性且在大规模强化学习中不稳定。我们提出SYNTHAGENT框架，联合生成多样化工具使用数据并模拟完整环境。具体而言，由强教师模型创建新颖任务与工具生态，再将其重写为有意模糊的指令，迫使智能体主动向用户查询缺失信息。处理合成任务时，基于LLM的用户模拟器提供用户私有信息，模拟工具系统则稳定返回工具响应。奖励机制基于任务级评分标准，涵盖子目标达成、用户-代理交互及禁止行为。在数学、搜索和工具使用等14个挑战性数据集上，使用合成数据训练的小模型表现显著提升，甚至超越更大规模的基线模型。
原文摘要 · Abstract (English)
Small LLMs often struggle to match the agentic capabilities of large, costly models. While reinforcement learning can help, progress has been limited by two structural bottlenecks: existing open-source agentic training data are narrow in task variety and easily solved; real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments. Specifically, a strong teacher model creates novel tasks and tool ecosystems, then rewrites them into intentionally underspecified instructions. This compels agents to actively query users for missing details. When handling synthetic tasks, an LLM-based user simulator provides user-private information, while a mock tool system delivers stable tool responses. For rewards, task-level rubrics are constructed based on required subgoals, user-agent interactions, and forbidden behaviors. Across 14 challenging datasets in math, search, and tool use, models trained on our synthetic data achieve substantial gains, with small models outperforming larger baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。