用更便宜的问答题替代生成题，快速评估大模型能力
From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models
- 将生成式任务转为选择题形式，降低计算开销
- 8个模型4项能力测试显示相关性高，平均提速35倍以上
- 适合训练中频繁评估模型性能的研究者
大模型训练过程中迭代评估至关重要，但生成式评估（NLG）耗时耗力。虽然自然语言理解（NLU）任务因固定答案选项而成本较低，但数学推理、代码生成等关键能力仍依赖耗时的生成式评估。本文将生成任务重构为计算成本更低的NLU形式，使用8个不同规模的模型，在数学推理、代码生成、事实知识和阅读理解4项能力上验证原任务与重构任务的相关性。结果表明二者高度相关，支持通过低成本方式评估核心能力，平均评估时间减少超过35倍。项目开源：https://github.com/Fraunhofer-IIS/EvalShortcut
原文摘要 · Abstract (English)
Iterative evaluation of LLMs during training is essential to ensure expected capability development, but can be time- and compute-intensive. While NLU tasks, where the model selects from fixed answer choices, are cheap to evaluate, essential capabilities like reasoning and code generation rely on the more time-consuming NLG (token-by-token generation) format. In this work, our aim is to decrease the computational burden of NLG benchmarks in order to enable monitoring crucial LLM capabilities during model training. We reformulate generative tasks into computationally cheaper NLU alternatives. We test the performance correlation between the original and reformulated tasks using 8 LMs of various sizes and 4 capabilities: mathematical reasoning, code generation, factual knowledge and reading comprehension. Our results show a strong correlation between task formats, supporting capability assessment via cheaper alternatives and achieving over 35x average reduction in evaluation time. Our project is available at: https://github.com/Fraunhofer-IIS/EvalShortcut
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。