提出可验证且省成本的LLM评估框架，减少测试样本量20%-40%
Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs
- 基于分块算法自适应选择测试样本，降低评估成本
- 理论推导出测试样本复杂度上界，确保结果可靠
- 在多个基准上节省20%-40%样本，95%置信度下误差可控
随着基础模型规模持续增长，训练模型的大小呈指数级扩张，给评估带来巨大挑战。当前评估方法依赖不断扩大的数据集来衡量大语言模型（LLMs）性能，但缺乏对测试数据充足性或有信息量样本选择的系统分析与指导。本文提出一种可验证且成本高效的LLM评估框架。该框架适用于不同评估目标，并输出包含真实值高概率的置信区间。我们引入「测试样本复杂度」量化评估所需测试点数量，并推导出紧致的上界。基于此理论，设计了一种名为Cer-Eval的分块算法，可自适应选择测试点以最小化评估成本。实际实验表明，Cer-Eval在多个基准上可节省20%至40%的测试样本，同时保持与现有评估过程相当的估计误差，并提供95%置信保证。
原文摘要 · Abstract (English)
As foundation models continue to scale, the size of trained models grows exponentially, presenting significant challenges for their evaluation. Current evaluation practices involve curating increasingly large datasets to assess the performance of large language models (LLMs). However, there is a lack of systematic analysis and guidance on determining the sufficiency of test data or selecting informative samples for evaluation. This paper introduces a certifiable and cost-efficient evaluation framework for LLMs. Our framework adapts to different evaluation objectives and outputs confidence intervals that contain true values with high probability. We use ``test sample complexity'' to quantify the number of test points needed for a certifiable evaluation and derive tight bounds on test sample complexity. Based on the developed theory, we develop a partition-based algorithm, named Cer-Eval, that adaptively selects test points to minimize the cost of LLM evaluation. Real-world experiments demonstrate that Cer-Eval can save 20% to 40% test points across various benchmarks, while maintaining an estimation error level comparable to the current evaluation process and providing a 95% confidence guarantee.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。