无需标注数据，用隐变量模型自动选最合适的语言模型。
Smoothie: Label Free Language Model Routing
- 基于多个模型输出构建隐变量图模型，推断每样本最优模型
- 在14个任务中9个正确识别最佳模型，路由准确率提升最高10点
- 适合无标注数据场景，尤其适用于多任务应用中的模型调度
大型语言模型（LLMs）在涵盖多种任务的场景中应用日益广泛。已有研究表明，不同输入样本适合不同的模型。现有方法多依赖人工标注数据训练辅助模型进行路由选择，而本文提出Smoothie，一种无需标签数据的无监督路由方法。给定多个模型的输出，Smoothie 构建一个关于可观测输出与未知“真实”输出嵌入表示的隐变量图模型，通过该模型估计每个样本对应各模型的质量评分，并将样本路由至得分最高的模型。实验表明，Smoothie 的质量评分与真实模型表现高度相关，在14个任务中有9个能正确识别最优模型，且在路由性能上相比基线最高提升10个百分点。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。