用嵌入向量替代文本，让大模型排序更快10倍以上
MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
- 用文本+嵌入混合表示输入，大幅缩短上下文长度
- 在相同延迟下吞吐提升10倍，比全文本模型快75.9倍
- 适合需要高并发、低延迟的工业级搜索推荐系统
大型语言模型（LLM）在捕捉语义细微差别方面表现优异，因此在现代推荐与搜索系统中展现出出色的相关性排序能力。然而，在工业级延迟和吞吐要求下，其计算开销过高。特别是交叉编码器排序系统常因需同时处理用户、查询和物品信息而产生长上下文预填充负载。为此，我们提出 MixLM，一种新型基于 LLM 的排序框架，通过减少输入上下文长度显著提升系统吞吐，同时保持交叉编码器排序器的语义能力。与传统纯文本输入不同，MixLM 采用文本与嵌入标记的混合表示：将目录中所有物品编码为少量嵌入标记并存入近线缓存，推理时使用这些嵌入标记替代原始文本，使物品长度从数千个文本标记降至少数嵌入标记。我们在 LinkedIn 实际搜索应用中部署了 MixLM 框架，分享了训练流程与在线服务基础设施优化的洞察。在相同延迟预算下，性能与强基线持平，吞吐提升 10.0 倍，相较全文本 LLM 重排序器提升 75.9 倍。效率提升使得全流量部署基于 LLM 的搜索成为可能，线上 A/B 测试显示日活用户（DAU）显著提升 0.47%。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems. However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, cross-encoder ranking systems often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information. To this end, we propose MixLM, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of cross-encoder rankers. In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use mix-interaction, a mixture of text and embedding tokens to represent the input. Specifically, MixLM encodes all items in the catalog into a few embedding tokens and stores in a nearline cache. The encoded item descriptions are used during online inference, effectively reducing the item length from a few thousand text tokens to a few embedding tokens. We share insights from deploying our MixLM framework to a real-world search application at LinkedIn, including a detailed discussion of our training pipelines, as well as a thorough analysis of our online serving infrastructure optimization. With the same latency budget and on-par relevance metrics, MixLM increased throughput by 10.0x comparing with strong baselines, 75.9x over full-text LLM rerankers. The efficiency gains delivered by MixLM enabled full-traffic deployment of LLM-powered search, which resulted in a significant 0.47\% increase in Daily Active Users (DAU) in online A/B tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。