Blink让大模型推理彻底绕开CPU，性能提升数倍且抗干扰。
Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
- 把请求处理和调度移到SmartNIC与GPU协同完成，不再依赖主机CPU。
- 在无干扰下延迟降低8.47倍，吞吐提升2.1倍，能耗减少48.6%。
- 适合追求高吞吐、低延迟、抗干扰的生产级大模型部署场景。
大语言模型（LLM）推理正成为数据中心核心服务，但现有服务栈仍将主机CPU置于关键路径上，导致性能易受CPU干扰，限制应用共置并迫使预留大量CPU资源，造成大量算力浪费。本文提出Blink，一种端到端服务架构，通过将主机CPU从稳态推理路径中移除，将职责重新分配给SmartNIC和GPU。Blink将请求处理卸载至SmartNIC，利用RDMA直接将输入送入GPU内存，并以持久化GPU内核替代主机调度，实现批处理、调度和KV缓存管理无需CPU参与。与TensorRT-LLM、vLLM和SGLang相比，Blink即使单独运行也表现更优：预饱和状态下P99 TTFT降低最高达8.47倍，P99 TPOT降低最高3.40倍，解码吞吐最高提升2.1倍，每令牌能耗降低最多48.6%。在CPU干扰下，Blink性能稳定，而现有系统性能下降最高达两个数量级。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$, improving decode throughput by up to 2.1$\times$, and reducing energy per token by up to 48.6$\%$. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。