突破百万词长上下文推理瓶颈，实现近线性扩展。
Context Parallelism for Scalable Million-Token Inference
- 提出上下文并行机制，跨16节点128块H100 GPU实现高效长文本处理。
- 100万词上下文预填充仅需77秒，12.8万词仅3.8秒，效率达93%。
- 支持全预填充、持久化键值缓存等场景，适合大规模长文本应用。
我们提出面向长上下文大语言模型推理的上下文并行方法，在16个节点共128块H100 GPU上实现了长上下文预填充延迟的近线性扩展。该方法在Llama3 405B模型上实现100万词上下文预填充仅需77秒（并行效率93%，计算利用率63%），12.8万词预填充仅需3.8秒。我们设计了两种无损精确环形注意力变体：pass-KV与pass-Q，覆盖全预填充、持久化键值缓存预填充及解码等多种使用场景，性能达到当前最优水平。在通过RDMA与TCP互联的H100 GPU主机上进行基准测试，均展现出良好可扩展性，表明该方法可在中低带宽商用数据中心稳定运行。
原文摘要 · Abstract (English)
We present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method achieves 1M context prefill with Llama3 405B model in 77s (93% parallelization efficiency, 63% FLOPS utilization) and 128K context prefill in 3.8s. We develop two lossless exact ring attention variants: pass-KV and pass-Q to cover a wide range of use cases with the state-of-the-art performance: full prefill, persistent KV prefill and decode. Benchmarks on H100 GPU hosts inter-connected with RDMA and TCP both show similar scalability for long-context prefill, demonstrating that our method scales well using common commercial data center with medium-to-low inter-host bandwidth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。