让大模型回答问题时附带精准句子级引用，提升可信度。
LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
- 用现成大模型自动生成带精确引用的长文本问答数据
- 构建4.5万条高质量训练数据，训练出80亿和90亿参数模型
- 生成答案时能同时输出准确回答与细粒度引用，优于GPT-4o
尽管当前长上下文大语言模型在基于长文本回答用户问题方面表现出色，但其回答缺乏引用，导致用户难以验证，引发对其潜在幻觉的信任担忧。本文旨在使长上下文大模型能够生成带有细粒度句级引用的回答，提升其忠实性与可验证性。我们首先提出LongBench-Cite，一个自动化基准测试，用于评估现有模型在长文本问答带引用（LQAC）任务上的表现，揭示了显著的改进空间。为此，我们提出CoF（粗到精）新流程，利用现成的大模型自动构建包含精确句级引用的长文本问答样本，并据此构建LongCite-45k大规模SFT数据集。最终，我们使用该数据集训练出LongCite-8B与LongCite-9B模型，成功实现单次输出中生成准确回答与细粒度句级引用。在LongBench-Cite上的评估显示，我们的模型在引用质量上达到领先水平，超越包括GPT-4o在内的先进闭源模型。
原文摘要 · Abstract (English)
Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering user questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to their potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations, improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs' performance in Long-Context Question Answering with Citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically generate long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the LongCite-45k dataset, successfully enabling their generation of accurate responses and fine-grained sentence-level citations in a single output. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。