为提示优化设计了不确定性评估基准，发现现有指标不匹配正确性判断需求。
Benchmarking Large Language Model Uncertainty for Prompt Optimization
- 构建多维度不确定性评估数据集，涵盖答案、正确性、随机与认知不确定性。
- 实验显示当前指标更反映输出信心而非正确性置信度。
- 适合关注大模型推理可靠性与提示工程优化的研究者参考。
大型语言模型（LLM）的提示优化算法在多步推理中表现优异，但仍缺乏有效的不确定性估计。本文提出一个基准数据集，用于评估答案、正确性、随机性和认知不确定性等度量指标。通过对 GPT-3.5-Turbo 与 Meta-Llama-3.1-8B-Instruct 等模型的分析，发现现有度量主要与答案不确定性相关，即反映输出的置信度和多样性，而非正确性不确定性。这表明需要开发更契合优化目标的不确定性度量，以更好指导提示优化。代码与数据集已开源。
原文摘要 · Abstract (English)
Prompt optimization algorithms for Large Language Models (LLMs) excel in multi-step reasoning but still lack effective uncertainty estimation. This paper introduces a benchmark dataset to evaluate uncertainty metrics, focusing on Answer, Correctness, Aleatoric, and Epistemic Uncertainty. Through analysis of models like GPT-3.5-Turbo and Meta-Llama-3.1-8B-Instruct, we show that current metrics align more with Answer Uncertainty, which reflects output confidence and diversity, rather than Correctness Uncertainty, highlighting the need for improved metrics that are optimization-objective-aware to better guide prompt optimization. Our code and dataset are available at https://github.com/0Frett/PO-Uncertainty-Benchmarking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。