通过分析输出分布特征，提升文本生成模型的置信度校准效果。
Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristics
- 基于输出概率分布设计无任务依赖的置信度评估方法。
- 在摘要、翻译和问答任务上显著改善BART与Flan-T5的置信度校准。
- 无需微调或启发式规则，适合各类文本生成场景使用。
可靠的置信度评分能提升文本生成模型的实际价值，例如引导用户重点关注低置信度预测，避免错误或危险输出。然而，文本生成中的置信度指标常未充分校准。原因之一是生成过程可能存在多个合理答案，而现有方法未能充分考虑此特性。当模型对多个有效序列分配较高概率时，即使整体置信度高，实际表现也可能不佳。为此，我们提出一种不依赖具体任务的置信度度量方法，仅基于模型输出的概率分布，无需额外微调或启发式规则。该方法在 summarization、translation 和 QA 数据集上有效提升了 BART 与 Flan-T5 的置信度校准效果。
原文摘要 · Abstract (English)
Well-calibrated model confidence scores can improve the usefulness of text generation models. For example, users can be prompted to review predictions with low confidence scores, to prevent models from returning bad or potentially dangerous predictions. However, confidence metrics are not always well calibrated in text generation. One reason is that in generation, there can be many valid answers, which previous methods do not always account for. Hence, a confident model could distribute its output probability among multiple sequences because they are all valid. We propose task-agnostic confidence metrics suited to generation, which rely solely on the probabilities associated with the model outputs without the need for further fine-tuning or heuristics. Using these, we are able to improve the calibration of BART and Flan-T5 on summarization, translation, and QA datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。