用图结构量化大模型长文本生成中的不确定性，提升事实准确性。
Graph-based Uncertainty Metrics for Long-form Language Model Outputs
- 将生成内容与其中主张构建成二分图，用图中心性度量不确定性。
- 相比传统方法，图中心性使AUPRC平均提升6.8%，事实性提升2-4%。
- 适合需要高可信度长文本生成的场景，如医疗、法律等专业领域。
大型语言模型在文本生成方面取得显著进展，但仍存在幻觉问题，且对长文本生成结果进行细粒度不确定性估计仍具挑战。本文提出图不确定性（Graph Uncertainty）方法，将模型生成内容与其中主张的关系表示为二分图，并利用一系列图中心性指标估计主张级别的不确定性。在此框架下，基于自一致性概念的现有方法可视为使用度中心性作为不确定性度量，我们证明更复杂的中心性指标如接近中心性能持续提升主张级不确定性估计效果。此外，我们设计了基于不确定性的解码技术，结合图结构和不确定性评分，仅保留最可靠的主张以提高生成内容的事实性。相比现有方法，我们的图基不确定性度量在多种长文本生成设置下，平均使AUPRC提升6.8%；端到端系统在事实性上相较现有解码技术稳定提升2-4%，同时显著增强生成回答的信息量。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have significantly improved text generation capabilities, but these systems are still known to hallucinate, and granular uncertainty estimation for long-form LLM generations remains challenging. In this work, we propose Graph Uncertainty -- which represents the relationship between LLM generations and claims within them as a bipartite graph and estimates the claim-level uncertainty with a family of graph centrality metrics. Under this view, existing uncertainty estimation methods based on the concept of self-consistency can be viewed as using degree centrality as an uncertainty measure, and we show that more sophisticated alternatives such as closeness centrality provide consistent gains at claim-level uncertainty estimation. Moreover, we present uncertainty-aware decoding techniques that leverage both the graph structure and uncertainty estimates to improve the factuality of LLM generations by preserving only the most reliable claims. Compared to existing methods, our graph-based uncertainty metrics lead to an average of 6.8% relative gains on AUPRC across various long-form generation settings, and our end-to-end system provides consistent 2-4% gains in factuality over existing decoding techniques while significantly improving the informativeness of generated responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。