为生成模型输出提供可证明有效的小预测集，提升可靠性。
Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regression
- 将生成预测集转化为最小样本数的置信回归问题。
- 在代码与数学题数据集上，预测集大小比现有方法减少15%-30%。
- 适用于需要保证至少一个有效输出的应用场景，如代码生成。
针对黑箱深度生成模型（如大语言模型）在给定输入（如文本提示）时生成有效且紧凑预测集的问题，本文提出生成预测集（GPS）方法。预测集的有效性由用户定义的二值可接受函数决定，例如代码生成中要求集合内至少一个程序通过所有测试用例。GPS基于校准样本和对生成模型的黑箱访问，通过挖掘获得可接受输出所需的最少样本数分布结构，采用简洁的置信回归策略生成具有可证明保障的预测集。在多个代码与数学应用数据集上使用不同大语言模型的实验表明，相较于最先进方法，GPS能显著缩小预测集规模（平均减少15%-30%），同时保持高有效性。
原文摘要 · Abstract (English)
We consider the problem of generating valid and small prediction sets by sampling outputs (e.g., software code and natural language text) from a black-box deep generative model for a given input (e.g., textual prompt). The validity of a prediction set is determined by a user-defined binary admissibility function depending on the target application. For example, requiring at least one program in the set to pass all test cases in code generation application. To address this problem, we develop a simple and effective conformal inference algorithm referred to as Generative Prediction Sets (GPS). Given a set of calibration examples and black-box access to a deep generative model, GPS can generate prediction sets with provable guarantees. The key insight behind GPS is to exploit the inherent structure within the distribution over the minimum number of samples needed to obtain an admissible output to develop a simple conformal regression approach over the minimum number of samples. Experiments on multiple datasets for code and math word problems using different large language models demonstrate the efficacy of GPS over state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。