用合成数据训练的查询重写模型比人类更准捕捉用户意图。
Can Synthetic Query Rewrites Capture User Intent Better than Humans in Retrieval-Augmented Generation?
- 用GPT-4o生成带上下文的高质量合成查询作为训练数据
- 在TopiOCQA和QRECC上优于人工重写,提升检索与生成效果
- 适合需要大规模、低成本查询重写的RAG系统研发者
多轮检索增强生成系统常面临口语化省略和指代模糊的查询,传统方法依赖人工标注重写,但受限于表达能力与理解深度,人工重写常偏离真实系统需求,导致用户意图与系统响应脱节。我们发现,高质量合成查询能更好弥合这一差距,在检索与生成任务中表现优于人工重写。本文提出SynRewrite,一种基于合成数据的查询重写模型。通过GPT-4o结合对话历史、当前查询、正样本文档与答案生成合成重写数据,再用Flan-T5微调,并利用生成器反馈通过DPO算法进一步优化。在TopiOCQA与QRECC数据集上的实验表明,SynRewrite在检索与生成任务上均持续优于人工重写,证明合成重写可作为可扩展且高效的替代方案。
原文摘要 · Abstract (English)
Multi-turn RAG systems often face queries with colloquial omissions and ambiguous references, posing significant challenges for effective retrieval and generation. Traditional query rewriting relies on human annotators to clarify queries, but due to limitations in annotators' expressive ability and depth of understanding, manually rewritten queries often diverge from those needed in real-world RAG systems, resulting in a gap between user intent and system response. We observe that high-quality synthetic queries can better bridge this gap, achieving superior performance in both retrieval and generation compared to human rewrites. This raises an interesting question: Can rewriting models trained on synthetic queries better capture user intent than human annotators? In this paper, we propose SynRewrite, a synthetic data-driven query rewriting model to generate high-quality synthetic rewrites more aligned with user intent. To construct training data, we prompt GPT-4o with dialogue history, current queries, positive documents, and answers to synthesize high-quality rewrites. A Flan-T5 model is then finetuned on this dataset to map dialogue history and queries to synthetic rewrites. Finally, we further enhance the rewriter using the generator's feedback through the DPO algorithm to boost end-task performance. Experiments on TopiOCQA and QRECC datasets show that SynRewrite consistently outperforms human rewrites in both retrieval and generation tasks. Our results demonstrate that synthetic rewrites can serve as a scalable and effective alternative to human annotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。