arXiv:2510.11358cs.CLcs.AI2025-10中稿 · CIKM 2026被引 3

不同大模型需要的检索证据各不相同,好证据因模型而异。

LLM-Specific Utility for Retrieval-Augmented Generation

  • 为四款大模型构建专属优质检索片段基准
  • 同一段证据对不同模型效果差异显著,最优证据不可通用
  • 现有方法难捕捉模型特异性效用,需定制化证据选择

检索增强生成(RAG)通常以主题相关性为目标优化,但其效果最终取决于检索到的段落是否对大语言模型(LLM)生成正确完整答案具有实际价值。我们指出,这种效用具有模型特异性,而非普适,源于模型间知识、推理和利用证据能力的差异。本文将LLM特异性效用定义为:在提供段落后,目标LLM性能相对于无证据回答的提升。为系统研究该问题,我们在三个问答数据集(Natural Questions、TriviaQA、MS MARCO-FQA)上,为Qwen3-8B/14B/32B和Llama 3.1-8B四款模型构建了专属黄金效用段落基准。分析显示,效用段落高度依赖模型且不可迁移:每款模型在其专属证据下表现最佳,而为其他模型优化的证据始终次优。人工标注证据仍是强基线,但无法完全满足特定模型需求。我们进一步提出LLM特异性效用判断任务,并构建相应基准SpecUBench。实验表明,现有效用感知的选段与评分方法主要捕捉普适效用,难以可靠估计模型特异性效用。总体而言,研究揭示了当前效用感知检索的局限性,推动面向生成器的定制化证据选择以提升RAG性能。代码与数据集见https://github.com/Trustworthy-Information-Access/LLM_specific_utility。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) is typically optimized for topical relevance, yet its success ultimately depends on whether retrieved passages are useful for a large language model (LLM) to generate correct and complete answers. We argue that such utility is often LLM-specific rather than universal, due to differences in models' knowledge, reasoning, and ability to leverage evidence. We formalize LLM-specific utility as the performance improvement of a target LLM when a passage is provided, compared to answering without evidence. To systematically study LLM-specific utility, we construct a benchmark of LLM-specific gold utilitarian passages for four LLMs (Qwen3-8B/14B/32B and Llama 3.1-8B) on three QA datasets (Natural Questions, TriviaQA, and MS MARCO-FQA). Our analysis shows that utilitarian passages are model-dependent and non-transferable: each LLM performs best with its own utilitarian evidence, while evidence optimized for other LLMs is consistently suboptimal. Human-annotated evidence remains a strong general baseline but does not fully match individual LLM utility needs. We further introduce the LLM-specific utility judgment task and construct the corresponding benchmark, i.e., SpecUBench (LLM-Specific Utility Benchmark). Experiments show that existing utility-aware selection and scoring methods largely capture model-agnostic usefulness and struggle to reliably estimate LLM-specific utility. Overall, our findings highlight the limitations of current utility-aware retrieval and motivate generator-tailored evidence selection for improving RAG. Our code and datasets can be found at https://github.com/Trustworthy-Information-Access/LLM_specific_utility.

检索增强大模型证据选择效用评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。