压缩大模型时，多选题表现好但自由回答失败，暴露评测盲区。
The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
- 通过剪枝压缩模型，研究其在多选与开放生成中的表现差异。
- 高稀疏度剪枝下，模型多选正确率高，但自由生成常失败。
- 答案未消失，而是被压低，用束搜索等方法可恢复。
压缩大语言模型能降低内存占用和推理成本，但也可能引发标准基准测试无法发现的失效问题。剪枝后的模型在多选题上表现良好，但在开放生成任务中却无法回答相同问题。我们研究了剪枝带来的变化：是抹除了正确答案，还是使其难以作为首选输出？通过多语言问答任务，对比剪枝前后同一问题的表现，发现存在‘评测幻觉’。在高稀疏度剪枝（尤其是Wanda方法）下，模型在贪婪解码的开放生成中常失败，但多选评分仍能选出正确答案。这类仅识别错误中，答案通常并未消失，而是被降级——通过束搜索、采样或一个上下文示例即可重新出现。因此，多选题基准可能过度高估压缩模型的实际可用性，形成评估盲区。压缩模型应测试其生成能力，而不仅限于识别能力。
原文摘要 · Abstract (English)
Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。