高效评估可能改变模型结论,需全面验证其可靠性。
Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions

- 在七种效率策略下对比模型表现,覆盖批处理、量化等
- 大批次保持准确率差距小于0.35个百分点,能耗降低五组
- 小样本子集对选取内容敏感,需谨慎设计评估方案
高效评估改变了支持模型行为结论的测试流程,但很少检验这些结论在评估成本降低后是否依然稳定。本文通过在BBQ和BBQ-V上对三款密集型与专家混合模型进行七种条件(包括批处理、量化、基准缩减及其组合)的测试,比较了准确率、偏差严重性与普遍性、推理质量、子群行为、子集成员稳定性、运行时间及测量到的GPU能耗,以全基准BF16基线为参考。大批次可将准确率保持在基线0.35个百分点内,并显著降低五组模型-数据集场景下的能耗;INT8基本维持质量但能耗为基线的1.79至4.26倍;INT4导致更大且依赖模型与上下文的变化;缩减基准提供最一致的节省,但极小子集对所选条目极为敏感。因此,高效评估应被视为一种测量干预,其有效性需针对基准所支持的各类结论逐一验证。项目网站:https://vectorinstitute.github.io/sustainable-rai-evaluation/,代码开源:https://github.com/VectorInstitute/sustainable-rai-evaluation。
原文摘要 · Abstract (English)
Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26$\times$ baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is https://vectorinstitute.github.io/sustainable-rai-evaluation/ and the code is available at https://github.com/VectorInstitute/sustainable-rai-evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。