用强化学习让冻结大模型的工具系统自动优化，提升任务完成率。
A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain
- 以可读的固定动作空间控制模型外挂系统，用强化学习在线学习策略。
- 在三个任务域中实现90%以上成功率，同时降低延迟和错误率。
- 适合希望快速部署可验证大模型代理的组织使用。
生产级大模型代理正越来越多地由冻结模型与外部控制框架（如提示模板、工具集、记忆层、规划策略和验证策略）组成。尽管Meta-Harness和HyperAgents等2026年系统表明该框架可自我优化或重写，但需昂贵代码搜索或无约束自修改代码，难以审计且无法用于全黑箱API。本文提出更受限的方案：将框架视为小型、固定、人类可读的动作空间，采用经典高效强化学习（ε-贪心上下文带、REINFORCE）在线学习策略，奖励函数包含多目标指标（任务成功、验证器得分、策略合规性、成本、延迟及未支持声明惩罚）。我们以DSPy（Khattab et al., 2024）作为上下文组装器和最强非自适应基线（DSPy BootstrapFewShot静态提示），在三个可验证任务域——工具使用流程、代码生成（HumanEval）、多跳检索问答（HotpotQA）——以及两个模型提供方（本地Ollama模型与AWS Bedrock）上进行评估。我们发布控制系统的代码、跨领域可验证任务套件、训练过程中的完整轨迹与奖励分解日志，以及适用于新组织领域与验证设置的无厂商依赖部署方案。
原文摘要 · Abstract (English)
Production LLM agents are increasingly assembled from a frozen model wrapped in a harness: a prompt template, a tool set, a memory/retrieval layer, a planning strategy, and a verification policy. Two 2026 systems, Meta-Harness (Lee et al., 2026) and HyperAgents (Meta AI, 2026), show that this harness can itself be optimized or even self-rewritten by an agentic proposer -- at the cost of either an expensive code-search loop or unconstrained self-modifying code, neither of which is auditable or usable with a fully black-box model API. We take a narrower, more constrained position: treat the harness as a small, fixed, human-legible action space and learn a policy over it online with classic sample-efficient reinforcement learning (an $ε$-greedy contextual bandit and REINFORCE), scored against a multi-objective reward (task success, verifier score, policy compliance, cost, latency, and an unsupported-claim penalty). We instantiate this control system with DSPy (Khattab et al., 2024) as both the context assembler and the source of the strongest non-adaptive baseline (a DSPy BootstrapFewShot static prompt), and evaluate it across three verifiable task domains -- tool-use workflows, code generation (HumanEval), and multi-hop retrieval QA (HotpotQA) -- and two model providers (a local Ollama model and AWS Bedrock). We release the harness-control-system code, the cross-domain verifiable task suite, the full trajectory/reward-decomposition logs from training, and a provider-agnostic deployment recipe for applying this to a new organization's domain and verification setup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。