用低维向量高效表示大模型能力，支持快速比较与选型。
LOCUS: Low-Dimensional Model Embeddings for Efficient Model Exploration, Comparison, and Selection
- 通过注意力机制对查询编码和评分生成固定维度嵌入
- 仅需基线1/4.8样本量即可获得可靠嵌入结果
- 嵌入空间几何意义明确，适合模型对比与组合选择
大型语言模型（LLM）生态的迅猛发展使有效管理与利用海量动态模型变得愈发困难。我们提出LOCUS，一种通过确定性前向传播生成低维向量嵌入的方法，以紧凑方式表征模型在各类查询下的能力。该方法基于注意力机制，利用编码器模型对查询编码与评估分数进行处理，无需重训练即可无缝集成新模型或更新已有模型嵌入。此外，我们训练了一个正确性预测器，利用模型嵌入与查询编码，在未见查询上实现当前最优路由准确率。实验表明，LOCUS相比基线最多可减少4.8倍查询评估样本数，同时生成的信息丰富且鲁棒。学习到的嵌入空间具有几何意义：距离相近表示模型相似，支持一系列下游应用，包括模型比较、聚类、模型组合选择以及不可用模型的鲁棒代理。
原文摘要 · Abstract (English)
The rapidly growing ecosystem of Large Language Models (LLMs) makes it increasingly challenging to manage and utilize the vast and dynamic pool of models effectively. We propose LOCUS, a method that produces low-dimensional vector embeddings that compactly represent a language model's capabilities across queries. LOCUS is an attention-based approach that generates embeddings by a deterministic forward pass over query encodings and evaluation scores via an encoder model, enabling seamless incorporation of new models to the pool and refinement of existing model embeddings without having to perform any retraining. We additionally train a correctness predictor that uses model embeddings and query encodings to achieve state-of-the-art routing accuracy on unseen queries. Experiments show that LOCUS needs up to 4.8x fewer query evaluation samples than baselines to produce informative and robust embeddings. Moreover, the learned embedding space is geometrically meaningful: proximity reflects model similarity, enabling a range of downstream applications including model comparison and clustering, model portfolio selection, and resilient proxies of unavailable models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。