Arctic Inference 通过动态并行策略实现高效企业级大模型推理，兼顾速度与成本。
Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI
- 采用动态调整的 Shift Parallelism 策略应对真实流量变化
- 请求完成速度提升 3.4 倍，生成速度提升 1.75 倍，单卡每秒处理 160 万词元嵌入
- 适合需要高性价比、低延迟的企业 AI 推理场景
推理已成为主导的 AI 工作负载，但现有系统在延迟、吞吐量和成本之间存在权衡。Arctic Inference 是 Snowflake AI Research 开源的 vLLM 插件，提出 Shift Parallelism 动态并行策略，在集成推测解码、SwiftKV 计算压缩和优化嵌入推理的基础上，实现请求完成速度最高提升 3.4 倍、生成速度提升 1.75 倍，单 GPU 每秒可处理 160 万词元嵌入，优于各类延迟或吞吐优化部署方案。目前已用于 Snowflake Cortex AI，为企业的先进、低成本推理提供支持，并向社区开放。
原文摘要 · Abstract (English)
Inference is now the dominant AI workload, yet existing systems force trade-offs between latency, throughput, and cost. Arctic Inference, an open-source vLLM plugin from Snowflake AI Research, introduces Shift Parallelism, a dynamic parallelism strategy that adapts to real-world traffic while integrating speculative decoding, SwiftKV compute reduction, and optimized embedding inference. It achieves up to 3.4 times faster request completion, 1.75 times faster generation, and 1.6M tokens/sec per GPU for embeddings, outperforming both latency- and throughput-optimized deployments. Already powering Snowflake Cortex AI, Arctic Inference delivers state-of-the-art, cost-effective inference for enterprise AI and is now available to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。