用奖励机制动态调用大模型，降低推理成本。
Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance
- 小模型生成每个词时，由奖励分数判断是否需要调用大模型
- 实验显示云调用次数减少，响应质量基本不变
- 适合对成本敏感但又需高质量输出的部署场景
大型语言模型（LLMs）在自然语言处理任务中表现出色，但其部署带来高昂的计算和财务成本。相比之下，小型语言模型（SLMs）虽可部署在低成本边缘设备上，却难以达到大模型的性能。本文提出一种新型混合推理方法，结合两者优势，同时最大限度减少对昂贵云上大模型的依赖。不同于将整个查询路由至 SLM 或云 LLM 的现有方法，本文引入基于奖励的机制，在生成每个词时动态决定是否调用云上 LLM。具体而言，对 SLM 生成的每个词计算奖励分数，仅当分数低于阈值时，才调用云上 LLM 进行下个词的预测。该方法不仅显著降低云上请求流量，从而降低成本，还能通过调整阈值灵活控制输出质量。实验结果表明，本方法在几乎不影响整体响应质量的前提下，大幅减少对云上 LLM 的使用，为高性能语言模型的低成本部署提供可行方案。
原文摘要 · Abstract (English)
Large language models (LLMs) are known for their exceptional performance across a range of natural language processing tasks, but their deployment comes at a high computational and financial cost. On the other hand, smaller language models (SLMs), which can be deployed on lower-cost edge devices, struggle to match the performance of their larger counterparts. This paper presents a novel hybrid inference approach that leverages the strengths of both model types while minimizing reliance on costly cloud-based LLMs. Unlike existing methods that route entire queries to either an SLM or a cloud LLM, our approach introduces a reward-based mechanism to dynamically determine the involvement of the cloud LLM during token generation. Specifically, each token predicted by the SLM is evaluated against a reward score, and only when this score falls below a certain threshold is the cloud LLM consulted for assistance in the next token prediction. This method not only reduces the traffic to the cloud LLM, thereby lowering costs, but also allows for flexible control over response quality depending on the reward score threshold. Experimental results demonstrate that our approach significantly reduces cloud LLM usage with minimal impact on overall response quality, offering a cost-effective solution for deploying high-performance language models
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。