用少量模型模拟评估，大幅缩小大模型评测数据集，提速且精准。
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
- 基于少量校准模型的蒙特卡洛模拟，自动确定最小有效评测子集。
- 在边缘设备上评测速度提升1.7至8.1倍，误差不超过2.62个百分点。
- 无需额外预测模块，适合资源受限场景下的高效模型评估。
在多种模型变体（如量化、微调或部署专用）上评估大语言模型，需反复运行大型基准测试，每次在边缘硬件（如NPU）上耗时可达数十小时。现有子集选择方法虽能降低成本，但依赖大规模校准池或学习型预测层。本文提出MINCE（蒙特卡洛驱动的小型化评测），利用少量校准模型的逐项日志进行蒙特卡洛模拟，确定保证精度漂移可控的最小子集大小，并随机固定该规模子集，无需预测层。MINCE在BF16模型上将IFEVAL、MMLU和GSM8K的评测数据集分别缩减54%、89%和70%，最大漂移≤2.62个百分点，平均漂移为0.77–3.59个百分点；在保留模型的NPU上实现1.7–2.0倍的评测加速，同时比tinyBenchmarks表现更优（MMLU漂移低12倍，GSM8K低3.3倍），且仅需其57分之一的校准模型。
原文摘要 · Abstract (English)
Evaluating LLMs across many model variants -- quantized, fine-tuned, or deployment-specific -- requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing subset selection methods reduce this cost but depend on large calibration pools or learned prediction layers. We introduce MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed. MINCE reduces IFEVAL by 54\%, MMLU by 89\%, and GSM8K by 70\% with maximum drift $\leq$2.62\,pp on BF16 models and mean drift of 0.77--3.59\,pp on held-out NPU models, while delivering median GPU evaluation speedups of 2.7--8.1$\times$ and NPU evaluation speedups of 1.7--2.0$\times$. The method is robust to calibration pool size and achieves lower drift than tinyBenchmarks (12$\times$ lower on MMLU, 3.3$\times$ on GSM8K) while using 57$\times$ fewer calibration models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。