从原始交互轨迹中无监督训练智能体，让大模型学会用工具完成真实任务。
AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories
- 通过事后任务推断和互信息分解，从噪声轨迹中提取精准行为奖励信号。
- 在三个真实场景中，性能超越更大模型（+8.7准确率/+9.7得分）。
- 适合资源有限、无标注数据的现实应用，无需模拟器或迭代试错。
基于大语言模型的智能体通过工具调用接口被越来越多地应用于真实场景，但针对特定环境训练仍面临根本挑战：真实应用缺乏预定义任务或验证器，也无可靠仿真器，且大规模环境交互预算有限。本文提出 extbf{AgentBrew}，一种离线训练框架，仅需单批次原始交互轨迹即可学习有效的工具使用策略，无需任务验证器或迭代在线采样。智能体先在目标环境中探索并收集未筛选的原始轨迹数据。为从该噪声数据中提取训练信号，采用“事后任务推断”根据实际结果重构每条轨迹对应的指令，并利用点互信息（PMI）实现“基于互信息的信用分配”，将轨迹总信息分解为可加的逐动作信用值。这些信用值用于加权策略训练目标，增强有效行为，抑制无效操作。在三个真实世界MCP应用（GitHub、Notion、PostgreSQL）上，AgentBrew使Qwen3-32B平均提升+8.7准确率/+9.7得分，优于Qwen3-235B（+2.3/+4.4），并显著超过拒绝采样方法（+5.9/+10.3）。结果表明，细粒度离线学习可从传统过滤方法丢弃的原始轨迹中恢复有用监督信号。代码已开源。
原文摘要 · Abstract (English)
LLM-based agents are increasingly deployed in real-world applications through tool-use APIs, yet training them for specific environments remains fundamentally difficult: real-world applications provide no pre-defined tasks or verifiers, no faithful simulators, and limited budget for large-scale environment interaction. In this paper, we propose \textbf{AgentBrew}, an offline training framework that learns effective tool-use policies from a single batch of raw interaction trajectories, without task verifiers or iterative on-policy rollouts. The agent first explores the target environment to collect a raw trajectory corpus without quality filtering. To extract training signal from this noisy corpus, \emph{retrospective task inference} reconstructs an aligned instruction for each trajectory based on its actual outcome, and \emph{PMI-Based credit assignment} decomposes the trajectory's total information about the inferred instruction into additive per-action credits via pointwise mutual information (PMI). These credits weight the policy training objective, amplifying informative actions while suppressing ineffective ones. On three real-world MCP applications (GitHub, Notion, PostgreSQL), AgentBrew improves Qwen3-32B by +8.7 Acc / +9.7 Score on average, surpassing Qwen3-235B (+2.3 / +4.4) and outperforming rejection sampling (+5.9 / +10.3). These results demonstrate that fine-grained offline learning can recover useful supervision from raw trajectories that filtering-based approaches would discard. The code is available at https://github.com/alphatogo/AgentBrew
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。