仅用提示词和大模型估算文本互信息，无需训练。
PromptNCE: Conditional Probabilities and PMI Using Only LLMs and Contrastive Estimation Prompts
- 用对比学习提示构建条件概率估计框架，引入其他类别避免封闭集偏差。
- 在三个数据集上表现最佳，PMI相关性最高达0.78。
- 适用于低数据场景，如学生知识总结评分，可直接复用提示模板。
从文本中估算互信息通常需要训练特定任务的判别器，限制了其在低数据场景的应用。我们探讨是否可通过仅使用提示词与模型生成的概率，零样本估算点互信息（PMI）。构建了基于三个公开人类标注数据集的基准，评估五种基于提示的信息论估计方法。主要方法 PromptNCE 将条件概率估计转化为对比任务，并在候选集外显式引入“其他”类别，使模型能将概率质量分配至候选集之外，避免标准对比提示的封闭集归一化问题。PromptNCE 在所有三个数据集上均取得最优条件概率估计结果。对于完整 PMI 估计，发现标签基率估计是两个数据集的主要瓶颈，最佳方法在相关性上达到 0.78。还通过计算机科学教育案例展示了该方法在低数据场景下对学生成绩摘要的评分应用。代码与提示已开源。
原文摘要 · Abstract (English)
Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings. We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities. We construct a benchmark from three publicly available human-annotated datasets with ground-truth PMI, and evaluate five information-theoretic prompting-based estimators. Our main method, PromptNCE, frames conditional probability estimation as a contrastive task and augments the candidate set with an explicit OTHER category. The OTHER category allows the model to assign probability mass outside the candidate set, avoiding the closed-set normalization of standard contrastive prompts. PromptNCE gives the best conditional probability estimates on all three datasets. For full PMI, we find that estimating label base rates is the primary bottleneck on two of the three datasets, with the best methods reaching Spearman correlation up to 0.78. We also present a case study in computer science education showing how these estimators can be used to score student knowledge summaries in a low-data setting. We release our code and prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。