ROMA让手机端运行大模型更快更省电，靠专用存储和混合架构。
ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM
- 用只读内存存量化基础模型，用高速缓存存可调参数，提升效率
- 在芯片上完整运行4位30亿或2位80亿参数模型，生成速度超2万词/秒
- 适合需要本地化部署、低延迟大模型的移动设备和嵌入式场景
随着大语言模型展现出强大能力，将其部署于边缘设备变得日益重要，带来隐私保护与实时交互的优势。QLoRA已成为设备端大模型的标准方法，通过量化降低内存与计算开销，同时利用LoRA实现任务自适应。本文提出ROMA，一种基于只读内存的QLoRA加速器，采用混合存储架构：用ROM存放量化后的基础模型，用SRAM存放LoRA权重与键值缓存。核心洞察是量化基础模型稳定且收敛，适合只读存储；而LoRA模块可灵活适配新数据，无需更新基础模型。为降低ROM面积开销，我们提出新型B-ROM设计，并与计算单元融合成一体电路，高效利用芯片资源。ROMA可在芯片内完整存储4比特30亿参数或2比特80亿参数的LLaMA模型，无需外部内存即可实现超过20,000词/秒的生成速度。
原文摘要 · Abstract (English)
As large language models (LLMs) demonstrate powerful capabilities, deploying them on edge devices has become increasingly crucial, offering advantages in privacy and real-time interaction. QLoRA has emerged as the standard approach for on-device LLMs, leveraging quantized models to reduce memory and computational costs while utilizing LoRA for task-specific adaptability. In this work, we propose ROMA, a QLoRA accelerator with a hybrid storage architecture that uses ROM for quantized base models and SRAM for LoRA weights and KV cache. Our insight is that the quantized base model is stable and converged, making it well-suited for ROM storage. Meanwhile, LoRA modules offer the flexibility to adapt to new data without requiring updates to the base model. To further reduce the area cost of ROM, we introduce a novel B-ROM design and integrate it with the compute unit to form a fused cell for efficient use of chip resources. ROMA can effectively store both a 4-bit 3B and a 2-bit 8B LLaMA model entirely on-chip, achieving a notable generation speed exceeding 20,000 tokens/s without requiring external memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。