让大模型生成的答案更可信：分解事实并自动修正错误
Decomposing and Revising What Language Models Generate
- 将长答案拆解为小事实，提升检索相关证据的准确性
- 发现矛盾时自动修正子事实，平均性能超越顶尖方法14%以上
- 适合需要可解释性与高精度的问答系统研发者
在大语言模型问答中，归因至关重要。现有基于问题分解的方法使用长答案生成问题以检索相关文档，但生成的问题常不相关且不完整，导致信息丢失；同时无法有效聚合不同文档和段落中的证据片段。为此，我们提出新的事实分解框架FIDES（忠实上下文增强的事实分解与证据聚合），采用上下文增强的两阶段忠实分解方法，将长答案分解为子事实，再由检索器获取相关证据片段。若检索到的证据与子事实冲突，则相应子事实将被修正。最终根据原始句子聚合证据。在六个数据集上进行的大量评估显示，使用新提出的自动归因精确率指标Attr_{auto-P}，FIDES在GPT-3.5-turbo、Gemini和Llama 70B系列上平均性能超越现有最佳方法超过14%。
原文摘要 · Abstract (English)
Attribution is crucial in question answering (QA) with Large Language Models (LLMs).SOTA question decomposition-based approaches use long form answers to generate questions for retrieving related documents. However, the generated questions are often irrelevant and incomplete, resulting in a loss of facts in retrieval.These approaches also fail to aggregate evidence snippets from different documents and paragraphs. To tackle these problems, we propose a new fact decomposition-based framework called FIDES (\textit{faithful context enhanced fact decomposition and evidence aggregation}) for attributed QA. FIDES uses a contextually enhanced two-stage faithful decomposition method to decompose long form answers into sub-facts, which are then used by a retriever to retrieve related evidence snippets. If the retrieved evidence snippets conflict with the related sub-facts, such sub-facts will be revised accordingly. Finally, the evidence snippets are aggregated according to the original sentences.Extensive evaluation has been conducted with six datasets, with an additionally proposed new metric called $Attr_{auto-P}$ for evaluating the evidence precision. FIDES outperforms the SOTA methods by over 14\% in average with GPT-3.5-turbo, Gemini and Llama 70B series.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。