利用大模型推理空闲算力同步进行微调，提升效率且不耽误响应速度。
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
- 通过共享前向计算结构，将推理空闲资源转为微调算力。
- 在保持100%推理服务达标率下，微调吞吐量提升2.9倍。
- 无需额外硬件，适合需高效利用算力的生产环境。
大模型服务系统为应对峰值负载而预留大量GPU算力，当流量低于峰值时造成大量闲置。本文提出DeltaServe，一种与主机无关的共服务架构，将这些闲置推理算力转化为LoRA微调吞吐量，同时保证推理服务等级目标（SLO）不变。DeltaServe通过轻量级钩子接口集成至现有推理引擎，仅需支持多LoRA批处理。其利用推理预填充与LoRA微调前向过程的共享执行结构，并采用基于SLO感知的调度器，在确保推理有足够余量时才执行微调。调度器由离线校准、在线优化的CUDA图感知延迟模型驱动。我们在vLLM、SGLang和S-LoRA中集成DeltaServe。基于某公司真实生产流量数据，相较于LLMStation，DeltaServe在vLLM上实现2.9倍的微调吞吐量提升，且推理SLO达成率达100%（对比后者为85%）。相比vLLM+torchtune基线，微调吞吐量提升39%，且无需额外硬件，完全满足SLO要求。
原文摘要 · Abstract (English)
LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Company X, DeltaServe on vLLM delivers 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。