不用跑代码,靠提示词分布就能预测大模型代码生成表现。
Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark
- 基于重要性采样理论,仅用已有标注数据重加权估算新任务表现。
- 对4个CodeLlama模型在9个基准上测试,代码正确率预测误差平均1.1%。
- 适合快速评估提示工程效果或检测数据污染,节省大量评测成本。
随着大语言模型快速发展,代码生成已成为评估其能力的关键基准。然而现有基准面临两大挑战:(1) 构建高质量测试集和参考解的成本持续上升;(2) 数据泄露风险增加,影响基准评估的可靠性。本文提出BIS——一种以提示为中心的评估框架,可在无需执行生成代码的情况下,仅通过分析提示分布预测大模型在代码生成任务上的表现。该方法基于重要性采样理论,利用重要性加权自编码器对已有标注基准样本进行重加权,以估计在新未见基准上的性能。为稳定估计,引入权重截断策略并计算拟合分布的边际期望。BIS作为补充工具,在资源受限下支持基准开发与验证,可快速提供提示选择与污染评估反馈。我们在4个CodeLlama模型和9个多样化基准上进行了大规模实验,覆盖8000个评估点。结果表明,该框架在代码正确率预测上平均绝对误差仅为1.1%,最佳与最差误差分别为0.3%和1.9%;对pass@1指标也表现出良好泛化能力,平均绝对误差为2.15%。这些结果证明BIS具有高可靠性和广泛适用性,能显著降低代码相关任务的模型评测成本与人力投入。
原文摘要 · Abstract (English)
With the rapid advancement of large language models , code generation has become a key benchmark for evaluating LLM capabilities. However, existing benchmarks face two major challenges: (1) the escalating cost of constructing high-quality test suites and reference solutions, and (2) the increasing risk of data contamination, which undermines the reliability of benchmark-based evaluations. In this paper, we propose BIS, a prompt-centric evaluation framework that enables ground-truth-free prediction of LLM performance on code generation tasks. Rather than executing generated code, BIS estimates performance metrics by analyzing the prompt distribution alone. Built on importance sampling theory and implemented using Importance Weighted Autoencoders, our method reweights samples from existing annotated benchmarks to estimate performance on new, unseen benchmarks. To stabilize the estimation, we introduce weight truncation strategies and compute marginal expectations across the fitted distributions. BIS serves as a complementary tool that supports benchmark development and validation under constrained resources, offering actionable and quick feedback for prompt selection and contamination assessment. We conduct extensive experiments involving 8,000 evaluation points across 4 CodeLlama models and 9 diverse benchmarks. Our framework achieves an average absolute prediction error of 1.1% for code correctness scores, with best- and worst-case errors of 0.3% and 1.9%, respectively. It also generalizes well to other metrics, attaining average absolute errors of 2.15% for pass@1. These results demonstrate the reliability and broad applicability of BIS, which can significantly reduce the cost and effort of benchmarking LLMs in code-related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。