测试深度研究智能体在多轮反馈下的改进能力,发现其难持续提升。
Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

- 设计新方法RGI,从评分标准推断研究过程漏洞
- 单轮过程反馈使得分提升8-15分,正确率提高35%-40%
- 后续轮次会倒退24%已达标项,难以持续改进
现有深度研究智能体(DRA)评估仅关注单次输出,忽略关键问题:智能体能否在反馈引导下改进报告?为此,我们开展多轮评估,对比两种反馈机制:自我反思(无外部诊断信号)与过程级反馈(针对研究策略漏洞的指导)。为实现过程级反馈,我们提出研究缺口推断(RGI),通过分析满足与未满足的评分标准模式,识别研究过程缺陷。结果揭示三点:(i) 自我反思下,智能体对评分标准的采纳与回退几乎持平,净提升可忽略;(ii) 单轮过程反馈带来显著提升,标准化得分增加约8-15点,采纳率达35%-40%;(iii) 后续轮次无法持续增益,重写报告时最多有24%的先前达标项被放弃。即便获得精准指导,当前架构仍难以实现可靠多轮改进。代码与结果公开于https://github.com/sabharwalrishabh/Multi-Turn-Evaluation-of-DRAs。
原文摘要 · Abstract (English)
Existing benchmarks for deep research agents (DRAs) assess only single-shot outputs, ignoring a key question: can DRAs improve their reports when guided by feedback? To investigate this, we conduct a multi-turn evaluation of DRAs under two feedback settings: self-reflection, in which the agent revises its report without any external diagnostic signal, and process-level feedback, in which the agent receives guidance targeting gaps in its research strategy. To enable process-level feedback, we design Research Gap Inference (RGI), a method that analyzes patterns of satisfied and unsatisfied rubric criteria to infer research-process gaps. Our analysis reveals three key findings: (i) under self-reflection, agents incorporate and regress on rubric criteria at nearly equal rates, yielding negligible net improvement; (ii) a single round of process-level feedback yields substantial gains, raising the normalized score by approximately $8$-$15$ points and yielding a roughly $35$-$40\%$ incorporation rate; (iii) these gains do not compound over subsequent turns, as agents regress on up to $24\%$ of previously satisfied criteria when rewriting the full report to address remaining gaps. Even with targeted guidance, reliable multi-turn improvement remains out of reach for the DRA architectures we evaluate. Our code and results are publicly available at https://github.com/sabharwalrishabh/Multi-Turn-Evaluation-of-DRAs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。