让大模型评估随提示变化，精准反映不同场景下的真实表现
Prompt-to-Leaderboard
- 用自然语言提示训练模型预测偏好系数，生成按提示定制的排行榜
- 在Chatbot Arena上实现排名第一的智能路由系统，性能优于平均榜单
- 揭示模型评估能力与提示相关性的幂律规律，适合个性化部署和评测
大型语言模型评估通常依赖于准确率或人类偏好等汇总指标，在用户和提示间取平均，掩盖了性能的个体差异。为此，我们提出提示到排行榜（P2L）方法，生成针对特定提示的排行榜。核心思想是训练一个大模型，以自然语言提示为输入，输出布拉德利-特瑞系数向量，用于预测人类偏好投票。由此产生的提示相关排行榜可实现无监督的任务特定评估、查询最优路由、个性化推荐以及模型优劣势的自动化分析。来自Chatbot Arena的数据表明，相较于平均排行榜，P2L更能捕捉语言模型性能的细微差异。此外，我们的研究发现，P2L生成提示特定评估的能力遵循与大模型自身类似的幂律缩放规律。2025年1月，基于该方法训练的路由系统在Chatbot Arena排行榜中位列第一。代码已开源：https://github.com/lmarena/p2l。
原文摘要 · Abstract (English)
Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leaderboard. Our code is available on GitHub at https://github.com/lmarena/p2l.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。