用大模型自动生成并评估对话搜索中的澄清问题，提升检索效果。
AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs
- 通过提示工程自动生成多样澄清问题，无需人工模板。
- 用多模型模拟人类评判，准确评估问题与答案质量。
- 生成问题比人工更优，显著提升BM25和交叉编码器检索效果。
在开放域对话搜索系统中，生成多样化且有效的澄清问题是提升查询理解与检索性能的关键。本文提出AGENT-CQ（自动生成与评估澄清问题），一个端到端的大语言模型框架，解决现有依赖人工标注或模板方法面临的可扩展性与适应性难题。AGENT-CQ包含两阶段：第一阶段采用大模型提示策略生成澄清问题；第二阶段（CrowdLLM）利用多个大模型实例模拟人类众包判断，基于综合质量指标评估生成问题与回答。在ClariQ数据集上的大量实验表明，CrowdLLM能有效评估问题与回答质量。人类评估与CrowdLLM均显示，AGENT-CQ生成阶段在问题与回答质量上持续优于基线。在基于检索的评估中，大模型生成的问题相比人工生成问题，显著提升了BM25与交叉编码器模型的检索效果。
原文摘要 · Abstract (English)
Generating diverse and effective clarifying questions is crucial for improving query understanding and retrieval performance in open-domain conversational search (CS) systems. We propose AGENT-CQ (Automatic GENeration, and evaluaTion of Clarifying Questions), an end-to-end LLM-based framework addressing the challenges of scalability and adaptability faced by existing methods that rely on manual curation or template-based approaches. AGENT-CQ consists of two stages: a generation stage employing LLM prompting strategies to generate clarifying questions, and an evaluation stage (CrowdLLM) that simulates human crowdsourcing judgments using multiple LLM instances to assess generated questions and answers based on comprehensive quality metrics. Extensive experiments on the ClariQ dataset demonstrate CrowdLLM's effectiveness in evaluating question and answer quality. Human evaluation and CrowdLLM show that the AGENT-CQ - generation stage, consistently outperforms baselines in various aspects of question and answer quality. In retrieval-based evaluation, LLM-generated questions significantly enhance retrieval effectiveness for both BM25 and cross-encoder models compared to human-generated questions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。