无需标注数据，用少量无标签样本就能估算大模型服务表现。
Plug-and-Play Performance Estimation for LLM Services without Relying on Labeled Data
- 基于大模型输出的负对数似然和困惑度，构建性能估计特征。
- 在多个任务和模型上验证，估计误差比基线低15%以上。
- 适合需要快速选型或优化大模型服务的工程团队使用。
大语言模型（LLM）通过上下文学习（ICL）在未见过的任务上表现出色，但其效果受任务和上下文影响，服务质量不一致。直接评估每次调用的性能成本高，尤其依赖大量标注数据或模型内部信息。本文提出一种新方法，仅需少量无标签样本即可“即插即用”地估算LLM服务性能。研究发现，从模型调用中提取的负对数似然和困惑度是有效且显著的特征。基于这些特征，采用四种不同的元模型进行性能预测。在多个LLM服务和任务上对比无标签基线，结果表明该方法更准确。实验还应用于服务选型和优化场景，验证了其有效性。
原文摘要 · Abstract (English)
Large Language Model (LLM) services exhibit impressive capability on unlearned tasks leveraging only a few examples by in-context learning (ICL). However, the success of ICL varies depending on the task and context, leading to heterogeneous service quality. Directly estimating the performance of LLM services at each invocation can be laborious, especially requiring abundant labeled data or internal information within the LLM. This paper introduces a novel method to estimate the performance of LLM services across different tasks and contexts, which can be "plug-and-play" utilizing only a few unlabeled samples like ICL. Our findings suggest that the negative log-likelihood and perplexity derived from LLM service invocation can function as effective and significant features. Based on these features, we utilize four distinct meta-models to estimate the performance of LLM services. Our proposed method is compared against unlabeled estimation baselines across multiple LLM services and tasks. And it is experimentally applied to two scenarios, demonstrating its effectiveness in the selection and further optimization of LLM services.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。