用高效注意力技术提升大模型推理速度，降低内存占用。
FlashEVA: Accelerating LLM inference via Efficient Attention
- 引入EVA注意力机制，通过控制变量优化计算效率。
- 推理速度最高提升6.7倍，显存占用降低5倍。
- 支持灵活调节吞吐与精度平衡，适合部署场景。
Transformer模型在自然语言处理中取得突破性进展，展现出卓越的可扩展性。然而，其对完整上下文的内存依赖严重制约了推理效率。本文提出FlashEVA，一种基于控制变量的高效注意力实现，并展示如何微调Transformer以适配该注意力机制。该方法仅需15亿个词元即可完成微调，且在多种下游任务中保持有效性。实验表明，FlashEVA相比标准Transformer实现，推理吞吐量最高提升6.7倍，峰值GPU内存使用降低5倍。尽管在检索类任务中存在局限，但其通过可调超参数实现吞吐与精度的灵活权衡，适用于多样化的实际应用场景。本工作为高效可适应的Transformer推理模型迈出关键一步。
原文摘要 · Abstract (English)
Transformer models have revolutionized natural language processing, achieving state-of-the-art performance and demonstrating remarkable scalability. However, their memory demands, particularly due to maintaining full context in memory, pose significant challenges for inference. In this paper, we present FlashEVA, an efficient implementation of EVA (Efficient Attention via Control Variates), and demonstrate how to finetune transformers to adapt to FlashEVA attention. Our method enables fine-tuning of Transformer models with as few as 1.5B tokens while preserving effectiveness across various downstream tasks. Notably, FlashEVA achieves up to 6.7x higher throughput and 5x lower peak GPU memory usage during inference compared to standard Transformer implementations. Despite these improvements, we observe limitations in retrieval-focused tasks. Our implementation offers control over the trade-off between throughput and accuracy through adjustable hyperparameters, providing flexibility for diverse use cases. This work represents a significant step towards more efficient and adaptable Transformer-based models for inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。