首次系统分析编程代理如何遵守指令计划，发现计划质量远超有无之分。
From Plan to Action: How Well Do Agents Follow the Plan?
- 通过21,120条轨迹对比八种计划变体，量化代理对计划的遵循程度。
- 无计划时代理依赖训练中内化的不完整流程，导致任务失败率上升37%。
- 早期插入不匹配的解题阶段会降低性能，提示需适配模型内部策略。
智能体常被要求遵循特定任务计划以获得指导，但其实际遵守计划的程度尚不明确。若缺乏此类分析，便无法判断解决方案是源于正确的战略推理，还是数据污染或基准过拟合所致。本文首次对编程代理在计划遵守方面进行了系统性分析，涵盖在SWE-bench Verified和SWE-bench Pro上使用四种LLM生成的21,120条轨迹，考察八种计划变体的影响。无显式计划时，代理会回退到训练中内化的流程，这些流程往往不完整、过拟合或应用不一致。提供标准计划可提升问题解决率；周期性计划提醒能缓解计划违背并提高任务成功率。然而，低质量计划的表现甚至劣于无计划，且早期插入与模型内部策略不符的任务阶段反而会降低性能。研究呼吁采用新微调范式，教会模型遵从指令计划，而非将其编码为固定流程，从而实现自适应推理与行动。
原文摘要 · Abstract (English)
Agents are commonly instructed to follow a task-specific plan for guidance. However, it is unknown to what extent agents actually follow instructed plans. Without such an analysis, determining the extent agents comply with a given plan, it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliance in programming agents, examining 21,120 trajectories from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on internalized workflows during training, which are often incomplete, overfit, or inconsistently applied. Providing the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and improve task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, inserting additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model's internal problem-solving strategy. These findings call for fine-tuning paradigms that teach models to follow instructed plans, rather than encoding task-specific plans in them, so that they reason and act adaptively, rather than memorizing workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。