用轻量探路器加速大模型学习，让语言模型在非语言任务上表现更优
Language-based Trial and Error Falls Behind in the Era of Experience
- 用小型神经网络快速探索环境，为大模型提供训练数据
- 使30亿参数模型在新任务上平均得分达0.86，超越部分商用模型
- 显著降低计算开销，节省约60%显存消耗，适合资源有限场景
尽管大语言模型在基于语言的智能体任务中表现出色，但在未见过的非语言环境（如符号或空间任务）中的应用仍受限。以往研究归因于预训练与测试分布不匹配，本文指出主要瓶颈是探索成本过高：掌握这些任务需要大量试错，而参数庞大的语言模型在高维语义空间中难以承担此开销。为此，提出SCOUT（Sub-Scale Collaboration On Unseen Tasks）框架，将探索与利用解耦。使用轻量级‘探路者’（如小型MLP）以远超语言模型的速度和规模探测环境动态，收集轨迹后通过监督微调（SFT）引导大模型，并辅以多轮强化学习激活其潜在世界知识。实验证明，该方法使Qwen2.5-3B-Instruct模型在新任务上平均得分达0.86，显著优于Gemini-2.5-Pro（0.60），同时节省约60%的GPU小时消耗。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.g., symbolic or spatial tasks) remains limited. Previous work attributes this performance gap to the mismatch between the pretraining distribution and the testing distribution. In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration: mastering these tasks requires extensive trial-and-error, which is computationally unsustainable for parameter-heavy LLMs operating in a high dimensional semantic space. To address this, we propose SCOUT (Sub-Scale Collaboration On Unseen Tasks), a novel framework that decouples exploration from exploitation. We employ lightweight "scouts" (e.g., small MLPs) to probe environmental dynamics at a speed and scale far exceeding LLMs. The collected trajectories are utilized to bootstrap the LLM via Supervised Fine-Tuning (SFT), followed by multi-turn Reinforcement Learning (RL) to activate its latent world knowledge. Empirically, SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming proprietary models, including Gemini-2.5-Pro (0.60), while saving about 60% GPU hours consumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。