大规模评估大模型的分布式框架，兼顾速度与统计严谨性。
Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation
- 基于Apache Spark实现数据并行评估，支持百万级样本
- 所有指标均带置信区间，比较结果通过显著性检验
- 用缓存减少重复推理，支持快速迭代评估策略
大规模评估大语言模型仍是众多组织的实际瓶颈。现有框架在处理数千样本时表现良好，但在面对数十万甚至上百万样本时性能下降。这种情况常见于跨领域行为评估或全面回归测试。我们提出Spark-LLM-Eval，一个原生构建于Apache Spark的分布式评估框架。该系统将评估视为数据并行问题，将样本分片到各执行器并进行统计正确的聚合。除吞吐量外，我们强调统计严谨性：每个报告指标均包含自助法置信区间，模型比较采用适当的显著性检验（配对t检验、McNemar检验或威尔科克斯符号秩检验，视指标类型而定）。框架还通过基于Delta Lake的内容可寻址响应缓存解决评估成本问题，使在不重跑推理的前提下即可迭代修改评估指标。我们描述了系统架构、统计方法，并报告基准结果，显示其随集群规模线性扩展。框架及全部评估代码均已开源。
原文摘要 · Abstract (English)
Evaluating large language models at scale remains a practical bottleneck for many organizations. While existing evaluation frameworks work well for thousands of examples, they struggle when datasets grow to hundreds of thousands or millions of samples. This scale is common when assessing model behavior across diverse domains or conducting comprehensive regression testing. We present Spark-LLM-Eval, a distributed evaluation framework built natively on Apache Spark. The system treats evaluation as a data-parallel problem, partitioningexamplesacrossexecutorsandaggregatingresultswithproperstatistical accounting. Beyond raw throughput, we emphasize statistical rigor: every reported metric includes bootstrap confidence intervals, and model comparisons come with appropriate significance tests (paired t-tests, McNemar's test, or Wilcoxon signed-rank, depending on the metric type). The framework also addresses the cost problem inherent in LLM evaluation through content-addressable response caching backed by Delta Lake, which allows iterating on metric definitions without re-running inference. We describe the system architecture, the statistical methodology, and report benchmark results showing linear scaling with cluster size. The framework and all evaluation code are available as open source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。