让大模型在客户端高效运行，显著降低显存占用并提升速度。
Efficient, VRAM-Constrained xLM Inference on Clients
- 采用分层模型拆分与跨CPU/GPU调度，优化推理性能。
- 交互模式下首字延迟降低6.7倍，吞吐提升30倍，显存减少10倍。
- 适合部署在游戏推理和物理智能推理等实际产品中使用。
为推动客户端AI创新，亟需在客户端实现高精度大语言模型（LLMs）与视觉语言模型（VLMs）的高效、无损推理，统称为xLMs。为此，本文提出流水线分片（pipelined sharding），一种基于基准测试引导的CPU-GPU混合调度技术，可高效支持密集型与专家混合（MoE）LLMs的显存受限推理。该方法结合子层级模型分片、CPU卸载、流水线复制-计算及优先级显存张量布局，同时优化首次标记时间（TTFT）与每秒生成词数（TPS），并能灵活适应不同系统与推理条件。针对高效高精度的VLM推理，我们结合流水线分片与llama.cpp实现的三项成熟策略（统称VLMOpt），包括视觉张量CPU卸载、flash attention及视觉与语言模型显存重叠避免。这些改进将应用于NVIDIA两大重要产品——In-Game Inferencing SDK与Cosmos-Reason1（CR1）物理智能推理VLM的未来版本。严格评估显示：在交互模式下，LLM的TTFT最高提升6.7倍，TPS提升30倍；CR1显存需求下降10倍；批量模式下吞吐最高提升8.2倍，均相对于各自激进基线。论文已被2026年第九届MLSys会议（产业赛道）接收。代码与成果已公开于https://github.com/deepshnv/pipeshard-mlsys26-ae。
原文摘要 · Abstract (English)
To usher in the next round of client AI innovation, there is an urgent need to enable efficient, lossless inference of high-accuracy large language models (LLMs) and vision language models (VLMs), jointly referred to as xLMs, on client systems. To address this, we present pipelined sharding, a novel, benchmark-profile-guided CPU-GPU hybrid scheduling technique to achieve efficient, VRAM-constrained inference for both dense and mixture-of-experts (MoE) LLMs. Using a combination of model sharding at the sub-layer level, CPU offloading, pipelined copy-compute, and prioritized tensor placement in VRAM, it optimizes both time-to-first-token (TTFT) and tokens per second (TPS) metrics, while flexibly adapting to system and inference conditions. For efficient, high-accuracy VLM inference, we combine pipelined sharding with a llama$.$cpp implementation of three well-understood prior ideas (jointly called VLMOpt), namely, vision tensor CPU offloading, flash attention, and vision and language model VRAM overlap avoidance. These enhancements are targeted at improving client xLM inference in future releases of two important NVIDIA products - the In-Game Inferencing software development kit (IGI SDK) and the Cosmos-Reason1 (CR1) physical AI reasoning VLM. Highlights from our rigorous evaluation spanning multiple models and client systems include: for interactive use, TTFT improves by up to 6.7x and TPS by up to 30x for LLMs, and CR1 inference's VRAM demand is down by 10x, while in batched mode, throughput improves by up to 8.2x, all compared to their respective aggressive baselines. This paper is accepted at the 9th MLSys Conference (Industry Track), 2026. Code and artifact available at: https://github.com/deepshnv/pipeshard-mlsys26-ae
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。