用例子预测代码和提示的运行成功率，解决大模型提示不可靠问题
Predicting Performance of Symbolic and Prompt Programs with Examples
- 将程序执行结果建模为伯努利分布，用先验概率预测未见任务表现
- 发现符号程序成功率要么全对要么全错，提示程序则多数接近正确
- 提出RAP方法，通过检索相似任务构建先验，提升预测准确性
大语言模型提示广泛用于自然任务，但其表现不可靠——可能在少数测试用例上成功，却在实际部署时失败。本文研究性能预测：给定一个程序（如Python代码或LLM提示）及若干领域内样例，预测其在未见任务上的表现。我们采用简单的抛硬币模型，将每次执行的成功/失败视为伯努利随机变量，其成功概率即程序的真实性能。该模型中，性能仅依赖于：1）测试用例上的观测结果；2）性能的先验分布。我们从多样化程序与任务数据集中提取经验先验，发现符号程序（如Python）的表现呈现全有或全无特征，而提示程序具有分散的先验，存在大量近乎正确的程序。这一差异解释了为何少量成功测试可验证符号程序，却不足以保证提示程序可靠。基于此，我们提出RAP（Retrieved Approximate Prior）方法，从现有语料库中检索相似任务与提示程序，构建代理先验，并用于性能预测。实验表明，RAP取得良好效果。
原文摘要 · Abstract (English)
LLM prompting is widely used for naturally stated tasks, yet it is unreliable it may succeed on a few test cases but fail at deployment time. We study performance prediction: given a program, either symbolic (e.g. Python) or a prompt executed on an LLM, and a few in-domain examples, predict its performance on unseen tasks from the same domain. We use a simple coin-flip model, treating each pass/fail program execution as a Bernoulli random variable, whose success probability is the programs unknown performance. In this model, performance depends entirely on: 1) the observed execution outcomes on test cases, and 2) a prior over performances. We compile empirical performance priors from a corpus of diverse programs and tasks, and find that performance for symbolic programs (e.g., Python) are all or nothing, while prompt programs have a diffuse prior with many nearly-correct programs. This difference explains why a few passing tests can certify symbolic programs but not prompt programs. Building on this insight, we develop RAP (Retrieved Approximate Prior), which retrieves similar tasks and prompt programs from an existing corpus to construct a proxy prior, which is then used to predict performance. We show RAP achieves solid performances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。