让大模型精准提取任意长度的证据,提升长文本摘要可信度。
Unstructured Evidence Attribution for Long Context Query Focused Summarization
- 提出无固定粒度的证据抽取方法,支持任意长度文本片段。
- 在5个大模型、4个数据集上验证,证据相关性与一致性显著提升。
- 开源数据集SUnsET与生成代码,助力可信赖摘要研究。
大型语言模型虽能从长文本中生成连贯摘要,但若能提取并引用证据段落,可增强摘要可信度。现有工作多限定于固定粒度(如句子、段落)的证据引用,本文提出提取任意长度的非结构化证据,以获取更相关且一致的证据。我们发现现有系统难以准确复制和引用此类证据,且常出现“中间丢失”现象。为此,我们构建了基于新生成流程的合成数据集SUnsET,用于训练无固定粒度证据摘要任务。在5个大模型、4个涵盖人工撰写、合成、单文档与多文档的数据集上,经SUnsET微调的模型能更广泛地从上下文中提取证据,生成更具相关性与事实一致性的摘要,优于未微调及固定粒度基线。相关数据集与代码已公开。
原文摘要 · Abstract (English)
Large language models (LLMs) are capable of generating coherent summaries from very long contexts given a user query, and extracting and citing evidence spans helps improve the trustworthiness of these summaries. Whereas previous work has focused on evidence citation with fixed levels of granularity (e.g. sentence, paragraph, document, etc.), we propose to extract unstructured (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case. We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle". To help models perform this task, we create the Summaries with Unstructured Evidence Text dataset (SUnsET), a synthetic dataset generated using a novel pipeline, which can be used as training supervision for unstructured evidence summarization. We demonstrate across 5 LLMs and 4 datasets spanning human written, synthetic, single, and multi-document settings that LLMs adapted with SUnsET generate more relevant and factually consistent evidence with their summaries, extract evidence from more diverse locations in their context, and can generate more relevant and consistent summaries than baselines with no fine-tuning and fixed granularity evidence. We release SUnsET and our generation code to the public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。