arXiv:2508.12096cs.CLcs.AI2025-08

通过关键样本对比,快速判断大模型推理能力强弱。

STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples

  • 基于同架构不同参数模型的性能跃迁,筛选关键测试样本。
  • 在6个基准上验证,能准确反映模型真实能力排名。
  • 轻量高效,适合快速比较各类大模型能力差异。

随着大语言模型能力迅速提升,传统评估方法面临挑战:标准基准上的高分未必代表真实推理能力增强,且普遍存在对公开基准的过拟合问题,全量评估成本高昂。为此,本文提出结构化过渡评估方法(STEM),一种轻量、可解释的相对能力评估框架。STEM通过分析相同架构但参数规模不同的模型在多个基准上的性能一致性跃迁,识别出‘显著过渡样本’(STS),利用这些样本可有效估计未知模型的能力位置。以Qwen3模型族在6个多样化代表性基准上构建STS数据池,并验证其泛化能力。实验表明,STEM能可靠捕捉性能趋势,与真实模型能力排名高度一致。该方法为细粒度、不依赖架构的模型评估提供了实用且可扩展的解决方案。

原文摘要 · Abstract (English)

Evaluating large language models (LLMs) has become increasingly challenging as model capabilities advance rapidly. While recent models often achieve higher scores on standard benchmarks, these improvements do not consistently reflect enhanced real-world reasoning capabilities. Moreover, widespread overfitting to public benchmarks and the high computational cost of full evaluations have made it both expensive and less effective to distinguish meaningful differences between models. To address these challenges, we propose the \textbf{S}tructured \textbf{T}ransition \textbf{E}valuation \textbf{M}ethod (STEM), a lightweight and interpretable evaluation framework for efficiently estimating the relative capabilities of LLMs. STEM identifies \textit{significant transition samples} (STS) by analyzing consistent performance transitions among LLMs of the same architecture but varying parameter scales. These samples enable STEM to effectively estimate the capability position of an unknown model. Qwen3 model family is applied to construct the STS pool on six diverse and representative benchmarks. To assess generalizability. Experimental results indicate that STEM reliably captures performance trends, aligns with ground-truth rankings of model capability. These findings highlight STEM as a practical and scalable method for fine-grained, architecture-agnostic evaluation of LLMs.

模型评估大模型能力对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。