首个多语言事实幻觉评估基准，助力可靠大模型研发
MUCH: A Multilingual Claim Hallucination Benchmark
- 构建多语言事实幻觉评估数据集，支持真实场景下的可靠性测试
- 包含4873条跨语言样本，24组生成日志供白盒方法开发
- 新算法仅需0.2%生成时间即可精准分割事实，适合实时监控
事实级不确定性量化（UQ）是缓解大语言模型（LLMs）可靠性不足的有前景方法。我们提出MUCH，首个面向真实环境、可公平复现的事实级UQ评估基准，涵盖4,873个样本，覆盖英语、法语、西班牙语和德语四种欧洲语言，以及四种指令微调的开源权重LLM。与以往基准不同，我们提供每令牌24组生成日志，支持未来白盒方法开发而无需重新生成数据。此外，不同于依赖人工或大模型分割的方法，我们提出一种确定性算法，仅需0.2%的LLM生成时间即可完成事实分割，适用于实时监控。评估表明，当前方法在性能与效率方面仍有巨大提升空间。
原文摘要 · Abstract (English)
Claim-level Uncertainty Quantification (UQ) is a promising approach to mitigate the lack of reliability in Large Language Models (LLMs). We introduce MUCH, the first claim-level UQ benchmark designed for fair and reproducible evaluation of future methods under realistic conditions. It includes 4,873 samples across four European languages (English, French, Spanish, and German) and four instruction-tuned open-weight LLMs. Unlike prior claim-level benchmarks, we release 24 generation logits per token, facilitating the development of future white-box methods without re-generating data. Moreover, in contrast to previous benchmarks that rely on manual or LLM-based segmentation, we propose a new deterministic algorithm capable of segmenting claims using as little as 0.2% of the LLM generation time. This makes our segmentation approach suitable for real-time monitoring of LLM outputs, ensuring that MUCH evaluates UQ methods under realistic deployment constraints. Finally, our evaluations show that current methods still have substantial room for improvement in both performance and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。