DiSCo通过设备与服务器协同调度，显著降低大模型文本流服务延迟和成本。
DiSCo: Device-Server Collaborative LLM-Based Text Streaming Services
- 动态分配请求并迁移生成任务，在设备与服务器间智能切换
- 尾部首令牌延迟降低11%-52%，平均首令牌延迟降低6%-78%
- 适合追求低延迟、低成本的实时文本服务部署场景
大型语言模型在文本流服务中的广泛应用带来了高昂成本与用户体验（QoE）挑战，尤其在满足实时交互所需的首令牌时间（TTFT）和令牌间隔时间（TBT）方面。实际测量显示，纯服务器部署受最后一公里网络延迟与波动影响，而设备端推理受限于资源。我们提出DiSCo，一种设备-服务器协同调度系统，通过成本感知调度，利用设备端推理的可预测速度与服务器端灵活容量，动态分配请求，并引入细粒度令牌级迁移机制，保障迁移过程中令牌连续性。在真实负载测试中——涵盖OpenAI GPT、DeepSeek等商用服务及LLaMA3等开源部署——DiSCo在不同模型-设备配置下，将尾部TTFT降低11%-52%，平均TTFT降低6%-78%，同时通过迁移机制将服务成本降低最高达84%，且维持相近的QoE水平。
原文摘要 · Abstract (English)
The rapid rise of large language models (LLMs) in text streaming services has introduced significant cost and Quality of Experience (QoE) challenges in serving millions of daily requests, especially in meeting Time-To-First-Token (TTFT) and Time-Between-Token (TBT) requirements for real-time interactions. Our real-world measurements show that both server-based and on-device deployments struggle to meet diverse QoE demands: server deployments face high costs and last-hop issues (e.g., Internet latency and dynamics), while on-device LLM inference is constrained by resources. We introduce DiSCo, a device-server cooperative scheduler designed to optimize users' QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. DiSCo employs cost-aware scheduling, leveraging the predictable speed of on-device LLM inference with the flexible capacity of server-based inference to dispatch requests on the fly, while introducing a token-level migration mechanism to ensure consistent token delivery during migration. Evaluations on real-world workloads -- including commercial services like OpenAI GPT and DeepSeek, and open-source deployments such as LLaMA3 -- show that DiSCo can improve users' QoE by reducing tail TTFT (11-52\%) and mean TTFT (6-78\%) across different model-device configurations, while dramatically reducing serving costs by up to 84\% through its migration mechanism while maintaining comparable QoE levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。