研究大模型多语言置信度估计，发现英语主导且提出改进提示策略
MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models
- 构建多语言置信度评估基准，涵盖跨语言与特定语言任务
- 英语在跨语言任务中置信度表现显著优于其他语言
- 针对特定语言任务，用本地化提示可显著提升可靠性
大语言模型生成幻觉的现象引发对其可靠性的担忧，因此评估生成结果的置信度至关重要。然而，非英语语言的置信度估计研究仍不充分。本文提出MlingConf，对大语言模型在多语言环境下的置信度估计进行系统研究，涵盖语言无关（LA）和语言特定（LS）两类任务，探索不同任务下多语言置信度的表现及语言主导效应。基准包含四个经过人工校验的高质量多语言数据集用于LA任务，以及一个针对特定社会、文化与地理背景的LS任务数据集。实验表明，在LA任务中，英语在置信度估计上表现出显著的语言优势；而在LS任务中，使用与问题语言一致的提示能增强语言主导性。该现象启发了一种简单有效的原生语气提示策略：针对LS任务采用语言特定提示，显著提升大模型在特定语言任务中的可靠性和准确性。
原文摘要 · Abstract (English)
The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluate high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy on LS tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。