用分层架构提升大模型离线推理效率，大幅降低生成成本。
Glinthawk: A Two-Tiered Architecture for Offline LLM Inference
- 将注意力计算移至低配层，高配层专注模型运算
- 吞吐提升5.9倍，生成成本降低2.8倍，长序列达16.3倍提升
- 适合批量处理等对延迟不敏感的场景，容错网络延迟
我们提出Glinthawk，一种面向离线大语言模型推理的两层架构。通过将注意力机制迁移至低性能计算层（第二层），而高阶加速器（第一层）仅负责模型推理，实现了注意力键值缓存内存需求与模型权重的独立扩展，从而支持更大批处理规模并提高加速器利用率。基于NVIDIA T4 GPU与标准CPU虚拟机原型验证，相比分页注意力基线，吞吐提升5.9倍，生成成本降低2.8倍；在长序列下，吞吐提升达16.3倍，成本降低2.4倍。评估表明该架构可容忍中等网络延迟且性能损失极小，适用于延迟容忍、高吞吐的批量处理任务。原型代码已公开于https://github.com/microsoft/glinthawk。
原文摘要 · Abstract (English)
We introduce Glinthawk, an architecture for offline Large Language Model (LLM) inference. By leveraging a two-tiered structure, Glinthawk optimizes the utilization of the high-end accelerators ("Tier 1") by offloading the attention mechanism to lower-end compute tier ("Tier 2"). This separation allows the memory demand of the attention, known as the key-value cache, to scale independently from the model weights, enabling larger batch sizes and more efficient accelerator usage. Prototyped with NVIDIA T4 GPUs and standard CPU VMs, Glinthawk improves throughput by $5.9\times$ and reduces cost of generation by $2.8\times$, compared to paged attention baselines. For long sequence lengths, it achieves $16.3\times$ throughput improvement at $2.4\times$ less cost. Our evaluation shows that this architecture can tolerate moderate network latency with minimal performance degradation, making it highly effective for latency-tolerant, throughput-focused applications such as batch processing. The prototype is publicly available at https://github.com/microsoft/glinthawk.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。