利用大模型回答不一致性，提升多候选答案的通过率。
Leveraging LLM Inconsistency to Boost Pass@k Performance
- 设计变体生成代理，对同一任务生成多个输入变体并提交不同答案。
- 在APPS数据集上，Pass@k性能显著优于基线方法。
- 适用于代码与网络安全等领域的前沿模型，通用性强。
大型语言模型在多个领域表现出色，但在输入微小变化时表现不一致。本文不将此视为缺陷，而是提出一种新方法：利用模型不一致性来提升Pass@k性能。我们设计了一个名为'Variator'的智能体，能为给定任务生成k个变体，并为每个变体提交一个候选答案。该方法不依赖具体任务，兼容自由格式输入，适用范围广。理论上，我们通过概率模型分析了不一致性效应；实验上，其在APPS数据集上的表现优于基线。此外，即使在前沿推理模型中，不一致性仍在代码与网络安全领域持续存在，表明该方法对未来模型仍具适用性。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve impressive abilities in numerous domains, but exhibit inconsistent performance in response to minor input changes. Rather than view this as a drawback, in this paper we introduce a novel method for leveraging models' inconsistency to boost Pass@k performance. Specifically, we present a "Variator" agent that generates k variants of a given task and submits one candidate solution for each one. Our variant generation approach is applicable to a wide range of domains as it is task agnostic and compatible with free-form inputs. We demonstrate the efficacy of our agent theoretically using a probabilistic model of the inconsistency effect, and show empirically that it outperforms the baseline on the APPS dataset. Furthermore, we establish that inconsistency persists even in frontier reasoning models across coding and cybersecurity domains, suggesting our method is likely to remain relevant for future model generations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。