用离线强化学习优化大模型执行流程，让固定流程变智能控制。
Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning
- 将执行流程建模为马尔可夫决策过程，用轻量控制器选择执行动作。
- 在六类任务中提升验证行为，最大提升来自零售与数据库任务基准。
- 区分最终结果与流程可靠性，适合追求稳健执行的场景。
大型语言模型（LLM）代理通常通过调整提示、模型或手动工作流来改进，而其执行流程被视为固定基础设施。本文认为该流程本身是可学习的控制层。我们将其形式化为有限时域的Harness MDP，其中轻量级控制器选择结构化执行动作，而LLM执行器保持冻结。控制器基于离线回放数据，仅使用最终任务-评分奖励，通过优势加权回归进行训练。同时，我们将最终任务质量与事后评估的Harness成熟度分数分离，后者衡量流程是否遵循可靠执行模式，而非仅关注答案正确性。这种分离提供了有限缓冲区下的流程学习视角：最终质量提升需依赖高回报离线数据支持，而流程行为可在与优势加权动作一致时自由调整。在六个受控领域及两个公开基准适配器上，学习到的控制器持续改善验证行为，并选择性提升最终任务质量，尤其在适配后的tau-bench零售、AgentBench DB-Bench和校准结构验证器的编程任务中表现最佳。消融实验表明，这些增益无法由行为克隆或强制检查解释。结果表明，对于冻结的LLM代理，流程控制是可学习的层次，且离线支持限制了更优流程控制向更好最终答案的转化。
原文摘要 · Abstract (English)
Large language model (LLM) agents are usually improved by changing prompts, models, or hand-written workflows, while the execution harness around the model is treated as fixed infrastructure. We argue that this harness is itself a learnable control layer. We formalize harness operation as a finite-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen. The controller is trained from offline rollouts using advantage-weighted regression with only terminal task-rubric rewards. We also separate final task quality from a post-hoc Harness Maturity Score, which measures whether the harness follows reliable execution patterns rather than only whether the final answer is correct. This separation gives a finite-buffer view of harness learning: final-quality gains require high-return support in the offline buffer, while process behavior can shift whenever it aligns with advantage-weighted actions. Across six controlled domains and two public-benchmark adapters, the learned controller consistently improves verification behavior and selectively improves final task quality, with the largest gains on adapted tau-bench retail, adapted AgentBench DB-Bench, and coding with a calibrated structural verifier. Ablations against behavior cloning and Forced CHECK show that the gains are not explained by imitation or by simply adding checks. These results identify harness control as a learnable layer for frozen LLM agents, while showing that offline support limits when better process control becomes better final answers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。