让大模型根据系统负载动态调整输出长度，降低延迟和能耗
BeLLMan: Controlling LLM Congestion
- 通过实时反馈系统负载，动态控制大模型生成长度
- 高峰期延迟降低8倍，能耗减少25%，请求量提升19%
- 适合高并发场景下的大模型服务优化，提升系统稳定性
大语言模型应用对底层基础设施缺乏感知，自回归生成文本时无视系统负载，导致推理延迟飙升和用户体验下降。我们提出的首个控制器beLLMan，使大模型基础设施能够主动、渐进地向第一方应用发出信号，根据系统负载变化调节输出长度。在配备H100 GPU的真实测试平台上，beLLMan在摘要任务的拥堵期间，将端到端延迟降低最高达8倍，能耗减少25%，同时支持处理多出19%的请求。
原文摘要 · Abstract (English)
Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。