用指令微调大模型重写对话问题，显著提升搜索效果
Rewriting Conversational Utterances with Instructed Large Language Models
- 用人类反馈强化学习训练指令模型，直接响应用户需求
- 重写对话问题后，最高提升31.7%的Precision@1和25.2%的MRR
- 适合做对话搜索、智能客服系统优化的研究与开发者
近期研究表明，大型语言模型（LLMs）在问答、摘要、编程和翻译等自然语言处理任务上表现优异，部分结果已接近人类专家水平。其最突破性能力在于通过零样本或少样本提示完成任务。该能力被用于训练指令微调的LLMs，利用人类反馈强化学习引导模型直接遵循用户请求。本文研究指令微调大模型在对话场景中重写用户问题以提升对话搜索效果的能力。我们探索何种提示能生成最具信息量的重写语句，从而实现最佳检索性能。在公开的TREC CAST数据集上进行可复现实验，结果表明，使用指令微调模型重写对话语句，在MRR上提升25.2%，Precision@1提升31.7%，NDCG@3提升27%，Recall@500提升11.5%，优于当前最优技术。
原文摘要 · Abstract (English)
Many recent studies have shown the ability of large language models (LLMs) to achieve state-of-the-art performance on many NLP tasks, such as question answering, text summarization, coding, and translation. In some cases, the results provided by LLMs are on par with those of human experts. These models' most disruptive innovation is their ability to perform tasks via zero-shot or few-shot prompting. This capability has been successfully exploited to train instructed LLMs, where reinforcement learning with human feedback is used to guide the model to follow the user's requests directly. In this paper, we investigate the ability of instructed LLMs to improve conversational search effectiveness by rewriting user questions in a conversational setting. We study which prompts provide the most informative rewritten utterances that lead to the best retrieval performance. Reproducible experiments are conducted on publicly-available TREC CAST datasets. The results show that rewriting conversational utterances with instructed LLMs achieves significant improvements of up to 25.2% in MRR, 31.7% in Precision@1, 27% in NDCG@3, and 11.5% in Recall@500 over state-of-the-art techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。