PQR生成多样真实用户提问，高效发现问答代理的失效场景。
PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures
- 通过双模块迭代优化，模拟真实用户意图生成测试问题。
- 在电商问答代理上发现23%至78%更多无帮助响应。
- 适合评估大模型代理可靠性，尤其关注实际应用中的失败点。
评估基于大语言模型的智能体仍具挑战性，因识别有意义的失败案例通常需大量人工设计真实测试场景。现有工作主要聚焦于自动发现由恶意用户引发的代理失效，却忽略了真实用户意图下同样会触发失败的问题。本文提出PQR框架，不仅能针对特定目标（如帮助性、安全性）揭示代理失效，还能模拟真实用户意图。PQR通过两个互补模块的迭代交互运行：查询优化模块进行重写以探索多样问题变体；提示优化模块利用历史反馈推导出违反目标的新策略与真实性规则，进而生成既能触发失败又保持真实的查询。我们在电商问答代理上评估PQR，结果表明其发现的无帮助响应比以往方法多23%至78%，且生成的问题更具多样性和真实性。
原文摘要 · Abstract (English)
Evaluating LLM-based agents remains challenging because identifying meaningful failure cases often requires substantial human effort to design realistic test scenarios. Prior works primarily focus on automatically discovering agent failures induced by adversarial users, while overlooking queries with real user intents that also trigger agent failures. We introduce PQR, a framework that not only surfaces agent failures with respect to specific objectives (e.g., helpfulness, safety, etc.) but also resembles real users' intents. PQR operates through an iterative interaction between two complementary modules. The query refinement module performs rewrites to explore diverse query variations, while the prompt refinement module uses prior feedback to derive new objective-violating strategies and realism policies for refining prompts, which in turn generate failure-triggering yet realistic queries. We evaluate PQR on detecting an e-commerce QA agent's unhelpful responses. Our method uncovers 23% - 78% more unhelpful responses, and our generated queries are more diverse and realistic compared to previous methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。