用高速闪存扩展大模型推理内存,解决容量瓶颈问题。
FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

- 动态合并与流水线化闪存读取,提升带宽利用率。
- 将刷新操作移出关键路径,降低延迟影响。
- 专用于只读权重的简化闪存管理,适合大模型推理场景。
大模型推理日益受加速器内存容量制约,而非计算吞吐量。在单加速器或小节点系统中,封装内内存容量有限,限制了可部署模型规模。高带宽闪存(HBF)是一种新兴的3D堆叠NAND闪存技术,提供接近加速器的多太字节容量,有望作为存储大模型权重的容量层。然而,现有基于HBF的方案面临三大挑战:(1) 依赖粗粒度静态预取以隐藏微秒级闪存读取延迟,但难以最大化读取吞吐;(2) 将闪存管理任务(如刷新)暴露在加速器可见的关键推理路径上;(3) 忽略针对工作负载行为优化闪存管理机制的机会。为此,我们提出FLINT,一种面向容量可扩展大模型推理的工作负载驱动型HBF基础架构。FLINT引入三项机制:(1) 硬件突发缓冲控制器,动态合并并流水线化HBF读取,利用现有闪存缓冲区维持高带宽;(2) 虚拟平面刷新机制,通过低成本资源复制将刷新操作移出读取主路径;(3) 只读FTL,以紧凑映射表替代SSD级任意写支持,将逻辑权重块映射至物理HBF位置。
原文摘要 · Abstract (English)
LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。