提出星型注意力机制，高效处理长序列的LLM推理。
Star Attention: Efficient LLM Inference over Long Sequences
- 分两阶段：先分块局部计算，再全局聚合缓存
- 推理速度提升11倍，内存降低，准确率保持97%-100%
- 适合大规模长文本生成与低延迟部署场景
基于Transformer的大型语言模型（LLM）在长序列上的推理因自注意力机制的二次复杂度而成本高、速度慢。我们提出星型注意力（Star Attention），一种两阶段的块稀疏近似方法，通过将注意力分布在多个主机上并最小化通信开销来提升计算效率。第一阶段，上下文在各主机间以分块局部注意力并行处理；第二阶段，查询和响应标记通过序列全局注意力访问所有先前缓存的标记。星型注意力可无缝集成至大多数使用全局注意力训练的Transformer-based LLM，将内存需求和推理时间减少最多11倍，同时保持97%-100%的准确率。
原文摘要 · Abstract (English)
Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。