arXiv:2606.30840cs.AI2026-06被引 1

通过对比失败与成功案例,自动优化大模型检索问答提示。

Contrastive Reflection for Iterative Prompt Optimization

  • 基于失败与成功行为对比,定位问题并生成针对性修改建议。
  • 在HotpotQA上将准确率从51.4%提升至60.4%,减少错误回归。
  • 适合需要可解释性修复的智能检索系统开发者。

大型语言模型代理正日益成为信息检索的核心:它们生成检索查询、合成答案,并越来越多地充当评估工具。改进控制这些代理的提示符是一个优化问题,但在实际应用中更像调试。工程师需要知道哪个行为失败了,附近哪些行为仍有效,两者有何区别,以及提示符修改是否提升了未见数据上的表现而未引入新错误。我们提出对比反思(Contrastive Reflection)框架,用于代理式信息检索工作流中的迭代提示优化。该框架以任务为中心的质量定义为基础:问答代理暴露检索或推理轨迹,评分代理提供维度级得分和理由。这些结构化轨迹被用来识别错误锚定的行为片段,从同一区域添加附近的成功示例,并请求教师大模型提出针对性提示修改。只有在验证性能提升时才接受候选修改,可选地进行回归检查。我们在一个公开的HotpotQA检索增强型问答设置上实现了该框架,使用树状切片选择器,一次对比修复使保留准确率从51.4%提升至60.4%。仅失败样本和随机证据的变体提升较少且破坏更多原正确例子。轻量级指令比较显示,该方法接近现代提示优化器:MIPROv2达59.4%,GEPA为57.0%。结果是一个可解释的优化循环,旨在使提示修复更具可查性与验证驱动性。

原文摘要 · Abstract (English)

LLM agents are becoming central to information retrieval: they issue retrieval queries, synthesize answers, and increasingly serve as judges for IR evaluation. Improving the prompts that control these agents is an optimization problem, but in applied IR settings it often looks less like blind search and more like debugging. Engineers need to know which behavior failed, which nearby behavior still worked, what distinguishes the two, and whether a prompt edit improves held-out quality without introducing regressions. We present Contrastive Reflection, an iterative prompt-optimization framework for agentic IR workflows. The framework starts from a task-centric quality definition: QA agents expose retrieval or reasoning traces, and grading agents expose dimension-level scores and rationales. These structured traces are used to identify error-anchored behavioral slices, add nearby successful examples from the same region, and ask a Teacher LLM to propose a targeted prompt edit. Candidate edits are accepted only when validation performance improves, optionally subject to regression checks. We instantiate the framework with a tree-based slice selector, but the contribution is the contrastive reflection loop rather than the tree itself. On a public HotpotQA retrieval-augmented QA setup, one tree-selected contrastive repair improves held-out exact-match accuracy from 51.4% to 60.4%. Failure-only and random-evidence variants improve less and break more previously correct examples. A light instruction-only comparison places the method near modern prompt optimizers: MIPROv2 reaches 59.4% and GEPA 57.0%. The result is an interpretable optimization loop for IR agents, aimed at making prompt repair more inspectable and validation-driven.

提示优化检索增强可解释性LLM代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。