共享解码模块提升多大模型部署效率，降低资源浪费。
SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
- 拆分Transformer模型，仅微调预填充部分，冻结解码部分供多模型共享。
- 相比传统方案，每GPU吞吐提升2.0倍，输出延迟波动小于5%。
- 支持低比特解码，量化后速度再快45%，适合资源受限场景。
在多模型大语言模型服务中，由于模型间无法跨模型批处理，解码阶段常因内存限制导致GPU严重闲置，尤其在负载不均时更明显。我们提出SUN（Shared Use of Next-token Prediction），首个实现分布式多模型服务中解码执行共享的方法。SUN将Decoder-only Transformer分解为预填充模块和解码模块，仅对任务相关的预填充模块进行微调，使冻结的解码模块可在不同模型间共享。该设计支持与模型无关的解码路由策略，均衡分配解码请求以最大化利用率。在多种任务和模型族上，SUN性能接近全量微调，同时用更少的解码工作节点维持系统吞吐。尤其在相同条件下，相较传统解耦架构，每GPU吞吐最高提升2.0倍，且单令牌输出时间（TPOT）保持在5%以内。SUN天然支持低比特解码；量化版本QSUN在保持共享解码优势的同时，实现45%的速度提升，精度仍与SUN相当。
原文摘要 · Abstract (English)
In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。