提出KTC方法，快速量化大模型生成内容的可信度。
Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantification
- 用核函数融合语言模型分布与词汇矛盾度，计算每条主张的不确定性。
- 在多语言数据集上达到现有方法平均性能，高精度场景下更优。
- 纯CPU运行速度超当前最佳方法65倍，适合生产环境实时监控。
主张级不确定性量化（UQ）旨在通过评估大语言模型输出中每条主张的真实性来缓解其可靠性不足的问题。本文提出轻量级的核令牌矛盾（KTC）方法，在现实白盒条件下实现主张级UQ。KTC将生成过程中涉及的候选词表示为半正定核，整合了语言模型的条件分布与词项矛盾得分，并利用冯·诺依曼熵量化该核的不确定性。为估计词项矛盾，我们基于维基百科语料库的频率统计提出新方法。尽管仅使用CPU，KTC相比基于交叉编码器的最先进GPU方法提速超过8.2倍，相比同类纯CPU方法提速超过65倍。评估覆盖两个基准测试、四种欧洲语言及16种不同模型。KTC不仅达到现有方法的平均性能，还在高精度场景中表现更优。该方法兼具计算高效与准确性，使大模型输出的实时监控在生产环境中成为可能。
原文摘要 · Abstract (English)
Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of each claim in their outputs. We introduce Kernel Token Contradiction (KTC), a lightweight approach to compute claim-level UQ under realistic white-box conditions. KTC represents the candidate tokens involved in LLM generation as a positive semi-definite kernel that integrates both the LLM's conditional distribution and a token contradiction score. We then use the Von Neumann entropy to quantify the uncertainty of this kernel. To estimate token contradiction, we develop a new approach based on frequency statistics from the Wikipedia corpus. Although CPU-only, our approach achieves over an 8.2x speedup compared to state-of-the-art GPU-accelerated methods based on cross-encoders, and over a 65x speedup compared to CPU-only methods with comparable performance. Our evaluation spans two benchmarks across four European languages and 16 different models. KTC not only matches the average performance of existing methods but also outperforms them in high-precision regimes. This combination of computational efficiency and accuracy makes real-time monitoring of LLM outputs practical in production.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。