用低成本代理模型实现大模型高效测试，显著提升评估精度。
Scaling Up Active Testing to Large Language Models
- 用上下文学习构建廉价代理模型，无需更新且小于目标模型。
- 无需调用目标模型即可做出优质数据选择，评估精度远超随机采样。
- 提出自举误差估计器，可实时判断测试效果是否良好。
主动测试通过精心选择数据实现标签高效的模型评估，但计算成本高。本文发现若干降本策略，使主动测试可扩展至大语言模型。关键在于：用于引导数据采集的代理模型可通过上下文学习低成本构建，无需在主动测试循环中更新，且可小于目标模型；甚至无需使用目标模型进行预测即可做出有效数据选择。结果表明，相比随机采样，该方法能更准确评估大模型性能。此外，我们引入一种自举误差估计器，可作为单次运行中主动测试效果的有效指标。
原文摘要 · Abstract (English)
Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。