通过预测生成长度动态调度，显著提升大模型推理效率与稳定性。
STAR: Decode-Phase Rescheduling for LLM Inference
- 用模型隐藏状态预测输出长度，精度高且开销极小
- 动态调度降低延迟75.1%，吞吐量提升2.63倍
- 适合长文本生成、高并发推理场景使用
大语言模型推理已成为基础范式，但输出长度变化导致解码阶段严重负载不均，尤其在长输出推理任务中。现有系统如PD分离架构依赖静态预填充-解码调度，常因解码负载波动导致SLO违规和内存溢出。本文提出STAR，一种基于长度预测的解码重调度系统。核心贡献包括：(1) 一种轻量级、连续的原生大模型预测方法，利用模型隐藏状态建模剩余生成长度，精度显著提升（MAE降低49.42%），参数量减少93.28%）；(2) 解码阶段的动态均衡调度机制，融合实时与预测负载，使P99 TPOT降低75.1%，良好吞吐量提升2.63倍。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference has emerged as a fundamental paradigm, however, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose STAR, a decode rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with a dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 75.1% and achieving 2.63 times higher goodput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。