提升大模型下一个词预测的置信度，让结果既可靠又简洁。
Conformal Prediction Sets for Next-Token Prediction in Large Language Models: Balancing Coverage Guarantees with Set Efficiency
- 用语义掩码和温度调整缩小候选词范围，保持置信保证
- 实测覆盖率90%目标达成，平均候选集从847缩至4.3个词
- 适合高风险场景中需要可解释预测的AI应用
在高风险领域部署大语言模型需严格量化不确定性，但传统softmax概率常失准。本文系统研究自适应预测集（APS）在基于Transformer、词汇量超25万的大模型中的下一词预测应用。核心发现：朴素的合取预测虽有效覆盖，但预测集动辄数百词，信息量低。为此提出词汇感知合取预测（VACP），利用语义掩码与温度调整评分，在保证边际覆盖的前提下显著缩小有效预测空间。在Gemma-2B模型上，SQUAD与WikiText基准测试显示，VACP实现89.7%的实测覆盖率（目标90%），平均预测集大小由847降至4.3，效率提升197倍。提供理论分析并开源实现以供复现。
原文摘要 · Abstract (English)
Deploying large language models (LLMs) in high-stakes domains requires rigorous uncertainty quantification, yet standard softmax probabilities are often poorly calibrated. We present a systematic study of Adaptive Prediction Sets (APS) applied to next-token prediction in transformer-based models with large vocabularies (greater than 250,000 tokens). Our central contribution is the identification of a coverage-efficiency tradeoff: while naive conformal prediction achieves valid coverage, it produces prediction sets of hundreds of tokens, rendering them uninformative. We propose Vocabulary-Aware Conformal Prediction (VACP), a framework that leverages semantic masking and temperature-adjusted scoring to reduce the effective prediction space while provably maintaining marginal coverage. Experiments on Gemma-2B using SQUAD and WikiText benchmarks demonstrate that VACP achieves 89.7 percent empirical coverage (90 percent target) while reducing the mean prediction set size from 847 tokens to 4.3 tokens -- a 197x improvement in efficiency. We provide a theoretical analysis of vocabulary reduction and release our implementation for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。