用课本自动生成语言模型测评数据，省去人工标注
Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora
- 用文本作为基础，让大模型自动生成评测问题
- 生成的问题与人工标注相关性高达0.91（斯皮尔曼）
- 适合快速评估模型在新领域的能力，尤其开放问答
语言模型持续进步，响应质量与连贯性不断提升。由于训练数据规模达到互联网级别，模型很可能在训练中见过用户可能提出的问题。已有大量评估基准用于衡量模型质量、回应恰当性和推理能力。然而，构建这些基准所需的人力正迅速被模型规模所超越。为每个兴趣领域手动创建基准不切实际。为此，我们提出一种基于文档语料库的自动化事实型合成数据评估方法。该方法利用相同的大模型，仅以参考文档（如教科书）为输入，自动评估特定领域的知识掌握情况。生成式基准评估与人工标注问题高度一致，综合斯皮尔曼等级相关系数达0.91，基准评估皮尔逊准确率0.74（模型特异性达0.82）。该方法可生成选择题和开放题，用于诊断模型能力。我们将其应用于三份近期文档（含两份超过语言模型知识截止时间），发现Gemma-3模型在开放题上表现惊人出色。代码已开源：https://github.com/mmajurski/grounded-synth-lm-benchmark
原文摘要 · Abstract (English)
Language Models (LMs) continue to advance, improving response quality and coherence. Given Internet-scale training datasets, LMs have likely encountered much of what users may ask them to generate in some form during their training. A plethora of evaluation benchmarks have been constructed to assess model quality, response appropriateness, and reasoning capabilities. However, the human effort required for benchmark construction is rapidly being outpaced by the size and scope of the models under evaluation. Having humans build a benchmark for every possible domain of interest is impractical. Therefore, we propose a methodology for automating the construction of fact-based synthetic data model evaluations grounded in document populations. This work leverages the same LMs to evaluate domain-specific knowledge automatically, using only grounding documents (e.g., a textbook) as input. This generative benchmarking approach corresponds well with human curated questions producing an ensemble Spearman ranking correlation of $0.91$ and a benchmark evaluation Pearson accuracy correlation of $0.74$ (model specific $0.82$). This novel approach supports generating both multiple choice and open-ended synthetic data questions to gain diagnostic insight of LM capability. We apply this methodology to evaluate model performance on three recent documents (two post LM knowledge cutoff), discovering a surprisingly strong performance from Gemma-3 models on open-ended questions. Code is available at https://github.com/mmajurski/grounded-synth-lm-benchmark
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。