用反例反馈让智能体自我修正，提升正则表达式生成的纠错能力。
Counterexamples as Feedback for Agent Self-Correction
- 通过反例作为反馈，引导智能体多轮修正错误的正则表达式。
- 在30个任务中，90%的错误能在4轮内修复，远超零样本生成的17%。
- 适合关注智能体自纠错能力与鲁棒性评估的研究者使用。
单轮代码生成指标低估了部署智能体的核心能力：收到具体反馈后能否修复错误产物。本文提出A-CEGIS框架，利用反例作为反馈，评估自然语言到正则表达式的多轮优化过程。智能体提出正则表达式，确定性验证器在全匹配语义下检查，紧凑的误报或漏报反例引导下一轮修正。在30个NL-RX-Turk任务中，诊断性反例反馈使90%的任务在四轮内解决，而零样本生成仅17%，通用自纠正为27%，仅错误反馈为23%。完整诊断运行中，隐藏集所有任务均在最终轮解决，平均成功耗时2.7轮，经针对性探测后仍保持77%的鲁棒成功率。结果表明，A-CEGIS能有效衡量智能体跨轮次的改进效率，并提供超越原始测试集的实用鲁棒性检验。
原文摘要 · Abstract (English)
Single-turn code-generation metrics understate a central property of deployed agents: whether they can repair a wrong artifact after receiving concrete feedback. This paper presents A-CEGIS, a lightweight framework that uses counterexamples as feedback for evaluating multi-turn refinement in natural-language-to-regex synthesis. An agent proposes a regex, a deterministic oracle checks it under full-match semantics, and compact false-positive or false-negative witnesses guide the next turn. On 30 NL-RX-Turk tasks, diagnostic counterexample feedback solves 90\% of tasks within a four-turn ablation budget, compared with 17% for zero-shot generation, 27% for generic self-correction, and 23% for error-only feedback. In a full diagnostic run with hardening, all tasks are solved on the hidden set by the final turn, with mean time-to-success of 2.7 turns and robust success of 77% after targeted probing. These results show that A-CEGIS measures how efficiently an agent improves across turns while adding a practical robustness check beyond the original held-out cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。