AI训练代理能执行但不会主动调整策略,缺的是动态反思能力。
What is Missing from AI Post-Training AI: An Empirical Analysis

- 通过分析大量训练轨迹,发现代理从一开始就锁定策略
- 增加经验、人工指导或计算量均无法改变策略僵化问题
- 核心瓶颈是缺乏执行中自发重审策略的机制
大型语言模型(LLM)代理现已可端到端后训练。它们能写代码、启动训练、评估检查点并提升下游性能,预示着人工智能自进化可能。我们指出,当前图景混淆了两种能力:执行层面能力(在选定策略内迭代)与策略层面能力(随实验证据积累修正高层决策)。分析公开发布的大量后训练轨迹发现,不同任务中,代理的训练策略在初期即被固定,后续所有预算仅用于该策略内的局部调优。我们检验三种自然解释——经验缺失、指导缺失、推理不足——并逐步加强干预。大量实验表明:(1)经验驱动的框架普遍提升执行效果(GSM8K+12.6,HumanEval+40.8),但策略仍不变;(2)人工指导可有效改变初始策略,但一旦训练开始,代理又陷入局部调整循环;(3)额外推理算力在简单任务上有效,但在最难任务上几乎无增益。结论:代理缺失的并非经验、指导或推理算力,而是执行过程中自发重审策略的机制。
原文摘要 · Abstract (English)
Large language model (LLM) agents can now post-train an LLM end-to-end. They can write code, launch training, evaluate checkpoints, and improve downstream performance, raising the prospect of AI-for-AI. We argue that this picture conflates two distinct capabilities: execution-level capability, iterating within a selected training strategy; and strategy-level capability, revising the high-level judgment as experimental evidence accumulates. Analyzing a large corpus of publicly released post-training trajectories, we find that across different tasks, the agent's training strategy is locked in at the very beginning, and the entire remaining budget is spent on local adjustments within the selected strategy. We then examine three natural explanations--missing experience, missing guidance, and insufficient reasoning--with escalating interventions. Extensive experiments show that (1) an experience-driven scaffold improves execution across the board (+12.6 points on GSM8K and +40.8 on HumanEval) but leaves the strategy static; (2) human guidance effectively redirects the initial strategy, yet the agent falls back into local adjustment loops once training starts; and (3) additional inference compute pays off on easier tasks but yields almost no gain on the hardest one. In conclusion, what agents lack is neither experience, guidance, nor reasoning compute, but a mechanism for spontaneously reevaluating their strategy during execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。