对比搜索与大模型,发现信息可验证性与实用性存在权衡。
The Extractive-Abstractive Spectrum: Uncovering Verifiability Trade-offs in LLM Generations
- 提出提取-摘要谱系,连接搜索与大模型的中间状态
- 越抽象生成,用户感知有用性提升200%,但引用率下降50%
- 适合高可信场景的系统应侧重可验证性,非通用模型
在学术研究中,专家始终标注来源。尽管大语言模型(LLMs)擅长信息整合,却无法提供可靠引用,难以追溯信息源头。相比之下,搜索引擎直接提供带链接的原始片段,将信息整合任务交由用户完成。通过调查发现,用户在高风险查询中更倾向使用搜索引擎,因信息来源的可靠性比模型回答的便利性更重要。为此,本文提出提取-摘要谱系,将搜索与大模型视为两端,涵盖多个未被探索的中间状态。搜索是提取式:返回带链接的原文片段;大模型是摘要式:合成并逻辑转换训练及上下文中的信息,但缺乏可靠引用。定义五种运行点,在七种系统上评估四类真实问答场景:网页搜索、语言简化、多步推理与医疗建议。结果显示,随着输出越来越抽象,感知实用性最高提升200%,但正确引用句子比例最多下降50%,用户验证引用信息时间最多增加三倍。研究建议针对特定领域选择合适运行点,并为高实用性模型提供可验证支持路径。
原文摘要 · Abstract (English)
Across all fields of academic study, experts cite their sources when sharing information. While large language models (LLMs) excel at synthesizing information, they do not provide reliable citation to sources, making it difficult to trace and verify the origins of the information they present. In contrast, search engines make sources readily accessible to users and place the burden of synthesizing information on the user. Through a survey, we find that users prefer search engines over LLMs for high-stakes queries, where concerns regarding information provenance outweigh the perceived utility of LLM responses. To examine the interplay between verifiability and utility of information-sharing tools, we introduce the extractive-abstractive spectrum, in which search engines and LLMs are extreme endpoints encapsulating multiple unexplored intermediate operating points. Search engines are extractive because they respond to queries with snippets of sources with links (citations) to the original webpages. LLMs are abstractive because they address queries with answers that synthesize and logically transform relevant information from training and in-context sources without reliable citation. We define five operating points that span the extractive-abstractive spectrum and conduct human evaluations on seven systems across four diverse query distributions that reflect real-world QA settings: web search, language simplification, multi-step reasoning, and medical advice. As outputs become more abstractive, we find that perceived utility improves by as much as 200%, while the proportion of properly cited sentences decreases by as much as 50% and users take up to 3 times as long to verify cited information. Our findings recommend distinct operating points for domain-specific LLM systems and our failure analysis informs approaches to high-utility LLM systems that empower users to verify information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。