arXiv:2603.18009cs.CLcs.AI2026-03

提出首个基于首词的不确定性度量,提升大模型分类可靠性。

How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding

  • 用首词不确定性衡量机制,融合类别先验抑制高频噪声
  • 在多选任务中提升平均准确率6.03%,减少50.66%检索触发率
  • 适合需要低计算成本、高可靠性的实际部署场景

随着大语言模型在自然语言处理中的广泛应用,提示工程与检索增强生成(RAG)已成为提升复杂任务性能的主流方法。然而,模型自回归生成带来固有的输出不确定性。由于性能高度依赖提示设计,精确的不确定性测量对可靠提示优化至关重要。对于多类多项选择(理解)任务,传统基于输出概率的不确定性度量(如熵)同等对待所有类别,忽略了预训练语料中各类别的先验差异。这导致无法区分由先验引发的虚假置信与由上下文理解带来的真实确定性,造成置信度校准不佳。为此,我们提出对数尺度焦点不确定性(LSFU),一种受焦点损失启发的首词基础度量。LSFU将标签先验概率作为风险调节因子,抑制高频类别的噪声,强化低频长尾类别的风险感知,并通过动态加权机制统一测量尺度。基于LSFU,我们进一步提出不确定性校准提示优化框架(UCPOF),利用模型输出的首词选择高质量样例并动态优化提示。全面评估显示,UCPOF相比少样本基线平均准确率提升6.03%,优于始终开启的完整RAG方案5.75%的平均准确率,且平均检索触发率降低50.66%。通过仅对高不确定性样本自适应触发RAG,该框架显著降低计算成本,同时保持领先性能。

原文摘要 · Abstract (English)

With the widespread adoption of large language models (LLMs) in natural language processing, prompt engineering and retrieval-augmented generation (RAG) have become mainstream to enhance LLMs' performance on complex tasks. However, LLMs generate outputs autoregressively, leading to inevitable output uncertainty. Since model performance is highly sensitive to prompt design, precise uncertainty measurement is crucial for reliable prompt optimization. For multi-class multiple-choice (understanding) tasks, conventional uncertainty measures (e.g., entropy) based on output probabilities treat all classes equally and ignore class prior differences in pretraining corpora. This failure to distinguish spurious confidence (from priors) from true certainty (from contextual understanding) results in poor confidence calibration. To address this, we propose Log-Scale Focal Uncertainty (LSFU), a first-token-based metric inspired by focal loss. LSFU incorporates label prior probabilities as a risk-modulation factor to suppress noise from high-frequency classes and emphasize risk for low-frequency long-tail classes, with a dynamic weighting mechanism unifying the measurement scale. Based on LSFU, we further propose the uncertainty-calibrated prompt optimization framework (UCPOF), which leverages the first token of model outputs to select high-quality exemplars and dynamically optimize prompts. Comprehensive evaluations show UCPOF improves average accuracy by 6.03% over few-shot baselines, surpasses always-on full RAG by 5.75% in overall average accuracy, and reduces the average retrieval trigger rate by 50.66%. By adaptively triggering RAG only for high-uncertainty samples, our framework significantly lowers computational costs while maintaining state-of-the-art performance.

大模型提示优化不确定性RAG

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。