arXiv:2603.01209cs.AIcs.LG2026-03

训练时保留运行时状态,能显著提升智能体的效率与稳定性。

Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics

  • 通过生成隐藏约束的优化任务,控制训练中是否保留中间状态。
  • 保留状态可减少80%的变量缺失错误,降低3.5倍的token消耗。
  • 训练数据应与部署环境一致,避免因状态管理不匹配导致失效。

工具增强型大模型作为智能体,常在推理与可执行Python代码间交替进行,如CodeAct框架所示。部署时,这些智能体依赖跨步骤持续的运行时状态。然而,用于后训练的轨迹通常未体现解释器状态的管理方式。本文探讨状态持久性是仅运行时的辅助机制,还是影响智能体学习解释器使用的训练数据属性。研究将状态持久性设为训练阶段的变量,提出一种名为Opaque Knapsack的程序生成任务集,该任务具有部分可观测性,需通过预算内的工具调用揭示信息,迫使多轮控制流与迭代状态更新。在固定任务实例、提示、工具、模型和监督的前提下,生成仅在状态是否持久化上不同的匹配轨迹。对同一基础模型(Qwen3-8B)分别在两种轨迹上微调,并评估四种训练-运行组合。2×2交叉实验表明:状态持久性不影响最终解决方案质量,但显著影响效率与稳定性——在无状态运行时,持久化训练模型约80%的回合出现变量缺失错误;而在持久化运行时,无状态训练模型需冗余重推已保存状态,耗时约3.5倍。因此,解释器持久性应被视为智能体轨迹的一阶语义。训练数据应与部署运行时对齐,以提升效率并减少脆弱的训练-运行不匹配。

原文摘要 · Abstract (English)

Tool-augmented LLMs are increasingly deployed as agents that interleave natural-language reasoning with executable Python actions, as in CodeAct-style frameworks. In deployment, these agents rely on runtime state that persists across steps. By contrast, the traces used to post-train these models rarely encode how interpreter state is managed. We ask whether interpreter persistence is merely a runtime scaffold, or a property of the training data that shapes how agents learn to use the interpreter. We isolate state persistence as a training-time variable. We introduce Opaque Knapsack, a procedurally generated family of partially observable optimization tasks designed to prevent one-shot solutions. Item attributes and constraints are hidden behind budgeted tool calls, forcing multi-turn control flow and iterative state revision. Holding task instances, prompts, tools, model, and supervision fixed, we generate matched trajectories differing only in whether interpreter state persists across steps or resets after each action. We then fine-tune identical base models (Qwen3-8B) on each trace variant and evaluate all four train-runtime combinations. Our 2x2 cross-evaluation shows that interpreter persistence shapes how agents reach solutions, not whether they do: solution quality is statistically indistinguishable across conditions, but token cost and stability differ substantially. A persistent-trained model in a stateless runtime triggers missing-variable errors in roughly 80% of episodes; a stateless-trained model in a persistent runtime redundantly re-derives retained state, using roughly 3.5x more tokens. Interpreter persistence should be treated as a first-class semantic of agent traces. Aligning fine-tuning data with deployment runtimes improves efficiency and reduces brittle train-runtime mismatches.

智能体状态管理训练对齐效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。