测试发现大模型在改写问题后准确率下降，暴露其依赖表面形式的脆弱性。
LLMs Show Surface-Form Brittleness Under Paraphrase Stress Tests
- 用改写问题重新评估模型，检验泛化能力
- 原题与改写题准确率差距显著，最高达15%以上
- 适合关注模型鲁棒性和评测可靠性的研究者
大型语言模型（LLMs）的基准测试分数可能因记忆测试项或近似重复而被高估。本文提出一种简单协议，通过重新评估模型在基准问题改写版本上的表现来探测其泛化能力。使用Mistral-7B-Instruct和Qwen2.5-7B-Instruct，在ARC-Easy和ARC-Challenge数据集上测量原始问题与改写问题间的准确率差异。该流程控制解码过程，强制多选输出格式，并包含稳健的改写清洗步骤以保留语义。结果发现，改写导致显著的准确率下降，表明模型存在对表面形式的非平凡依赖，印证了先前关于数据污染和表面形式捷径的担忧。
原文摘要 · Abstract (English)
Benchmark scores for Large Language Models (LLMs) can be inflated by memorization of test items or near duplicates. We present a simple, protocol that probes generalization by re-evaluating models on paraphrased versions of benchmark questions. Using Mistral-7B-Instruct and Qwen2.5-7B-Instruct, we measure the accuracy gap between original and paraphrased items on ARC-Easy and ARC-Challenge. Our pipeline controls decoding, enforces multiple-choice output format, and includes a robust paraphrase-cleaning step to preserve semantics. We find that paraphrasing induces a non-trivial accuracy drop (original vs. paraphrased), consistent with prior concerns about contamination and brittle surface-form shortcuts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。