提出统一评估大模型一致性与鲁棒性的框架SCORE，揭示真实场景下性能波动问题。
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models
- 通过多轮不同设置测试同一基准，评估模型在真实场景下的稳定性。
- MMLU-Pro上提示改写导致准确率波动达10%，AGIEval中选项顺序变化致误差6.1%。
- 开源代码并建立鲁棒性排行榜，适合关注模型可靠性研究者使用。
大型语言模型（LLMs）的典型评估通常在特定条件下报告单一指标，仅反映最佳表现，忽视了实际应用中的鲁棒性与可靠性。例如，在MMLU-Pro数据集上，简单改写提示会导致准确率波动高达10%；在AGIEval数据集中，答案选项顺序调整可引发最大6.1%的准确率差异。尽管已有研究讨论模型鲁棒性问题，但缺乏统一、集中的评估框架。为填补这一空白并整合现有研究，我们提出SCORE（Systematic Consistency and Robustness Evaluation），一个针对非对抗性场景的全面评估框架。该框架通过在多种设置下反复测试模型于同一基准，提供对其准确率与一致性的更真实估计。我们公开发布代码，并启动大语言模型鲁棒性排行榜，以推动后续研究与发展。
原文摘要 · Abstract (English)
Typical evaluations of Large Language Models (LLMs) report a single metric per dataset, often representing the model's best-case performance under carefully selected settings. Unfortunately, this approach overlooks model robustness and reliability in real-world applications. For instance, simple paraphrasing of prompts on the MMLU-Pro dataset causes accuracy fluctuations of up to 10\%, while reordering answer choices in the AGIEval dataset results in accuracy differences of up to 6.1\%. While some studies discuss issues with LLM robustness, there is no unified or centralized framework for evaluating the robustness of language models. To address this gap and consolidate existing research on model robustness, we present SCORE ($\mathbf{S}$ystematic $\mathbf{CO}$nsistency and $\mathbf{R}$obustness $\mathbf{E}$valuation), a comprehensive framework for non-adversarial evaluation of LLMs. The SCORE framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. We release the code publicly and start an LLM robustness leaderboard to facilitate further development and research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。