给大模型修复错误提供结构化反馈，成功率提升近一倍。
Structured Feedback Improves Repair in an LLM Agent Loop
- 用结构化反馈明确失败位置、实际值和可接受选项，指导模型改进。
- 在50场游戏测试中，成功率从14%提升至36%（Qwen2.5-Coder-14B）。
- 适合需要高效自我修正的AI代理系统开发者参考。
LLM代理在外部验证拒绝候选方案后常会重试，但验证与下一轮模型调用之间的接口仍不明确。我们提出VeriHarness，一种由代码控制的代理循环，其中模型生成候选方案，外部验证器控制接受标准、预算和执行轨迹。通过该框架对比原始诊断与包含失败位置、观察值及可接受替代方案三要素的结构化反馈。在50个受限于四次调用的TextWorld游戏中，对Qwen2.5-Coder-14B，成功率从14/50提升至36/50（+44个百分点）；对Llama-3.1-8B，从8/50提升至29/50（+42个百分点）。消融实验表明，大部分增益来自可接受替代方案；仅提供位置和观察值的反馈仍接近原始诊断基线。以自然语言描述完整修复信息的效果与键值对JSON相当，无证据显示JSON格式本身促进修复。该优势在不同调用预算和采样解码设置下均保持一致。
原文摘要 · Abstract (English)
LLM agents often retry after external validation rejects a candidate, but the interface between validation and the next model call remains underspecified. We introduce VeriHarness, a code-controlled agent loop in which models generate candidates while external validators control acceptance, budgets, and traces. We use it to compare raw diagnostics with feedback that identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games under a four-call cap, feedback containing all three fields raises terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B (+44 percentage points) and from 8/50 to 29/50 for Llama-3.1-8B (+42 points). Ablations locate most of the gain in the admissible alternatives: feedback containing only the location and observed value remains near the raw diagnostic baseline. Presenting the complete repair information in prose instead of a keyed JSON record yields nearly the same success, providing no evidence that JSON syntax itself improves repair. The ordering persists across the tested call budgets and one sampled-decoding setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。