用大模型标注文档有用性，降低人工标注成本。
Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation
- 用大模型标注文档对问答任务的实用价值，替代人工标注。
- 仅用20%人工标签+大模型标注，效果接近全人工标注。
- 在多个数据集上提升检索与生成效果，适合新语料系统初始化。
本文探索使用大语言模型（LLMs）对文档实用价值进行标注，以减少训练检索和检索增强生成（RAG）系统对昂贵人工标注的依赖。针对检索相关性与生成实用性之间的差距，我们利用大模型标注文档实用价值。为有效利用每个查询的多个正样本，提出一种新损失函数，最大化其联合边际似然。采用Qwen-2.5-32B模型，在MS MARCO数据集上进行标注，并在MS MARCO、BEIR上进行检索实验，以及在MS MARCO QA、NQ、HotpotQA上进行RAG实验。结果表明，大模型生成的标注可提升跨领域检索性能，并优于仅使用人工标注或下游问答指标训练的模型。此外，结合20%人工标签与大模型标注，性能可媲美全量人工标注。本研究为在新语料上初始化问答系统提供了全面方法。
原文摘要 · Abstract (English)
This paper explores the use of large language models (LLMs) for annotating document utility in training retrieval and retrieval-augmented generation (RAG) systems, aiming to reduce dependence on costly human annotations. We address the gap between retrieval relevance and generative utility by employing LLMs to annotate document utility. To effectively utilize multiple positive samples per query, we introduce a novel loss that maximizes their summed marginal likelihood. Using the Qwen-2.5-32B model, we annotate utility on the MS MARCO dataset and conduct retrieval experiments on MS MARCO and BEIR, as well as RAG experiments on MS MARCO QA, NQ, and HotpotQA. Our results show that LLM-generated annotations enhance out-of-domain retrieval performance and improve RAG outcomes compared to models trained solely on human annotations or downstream QA metrics. Furthermore, combining LLM annotations with just 20% of human labels achieves performance comparable to using full human annotations. Our study offers a comprehensive approach to utilizing LLM annotations for initializing QA systems on new corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。