用激活神经元比例衡量大模型效率，发现性能越强越省力。
Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric
- 通过计算推理时激活的神经元比例，量化模型完成任务的投入成本。
- 发现模型性能与投入成本呈反向对数关系，形成通用的效用定律。
- 可用于训练诊断、数据污染检测和更公平的模型对比，适合评估者使用。
大语言模型在学术、产业和日常应用中日益重要，但现有评估方法难以跟上其快速发展。核心挑战在于泛化问题：如何从有限基准测试推断模型近乎无限的能力。本文提出模型利用率指数（MUI），一种基于机制可解释性的补充指标，通过计算推理时激活的神经元或特征比例来量化模型任务投入。直观上，真正强大的模型应以更低投入实现更高性能。对多个主流大模型的实验揭示，MUI与性能间存在稳定的反向对数关系，我们将其归纳为效用定律。由此衍生出四项实用推论：(i)指导训练诊断，(ii)暴露数据污染问题，(iii)实现更公平的模型比较，(iv)设计模型专属的数据集多样性。代码见 https://github.com/ALEX-nlp/MUI-Eva。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications, yet current evaluation methods struggle to keep pace with their rapid development. One core challenge of evaluation in the large language model (LLM) era is the generalization issue: how to infer a model's near-unbounded abilities from inevitably bounded benchmarks. We address this challenge by proposing Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores. MUI quantifies the effort a model expends on a task, defined as the proportion of activated neurons or features during inference. Intuitively, a truly capable model should achieve higher performance with lower effort. Extensive experiments across popular LLMs reveal a consistent inverse logarithmic relationship between MUI and performance, which we formulate as the Utility Law. From this law we derive four practical corollaries that (i) guide training diagnostics, (ii) expose data contamination issue, (iii) enable fairer model comparisons, and (iv) design model-specific dataset diversity. Our code can be found at https://github.com/ALEX-nlp/MUI-Eva.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。