边端大模型推理中，用双路径缓存优化内存不足问题。
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
- 根据内存情况动态选择缓存路径，避开文件系统开销。
- 预填充和解码延迟降低最高达42.4%，固态盘利用率提升2.2倍。
- 适合资源受限的边缘设备部署大模型推理任务。
大型语言模型（LLM）在边缘AI系统中的部署日益增多，但受限于内存容量，关键-值（KV）缓存常超出设备可用内存。尽管基于NVMe的卸载可提供扩展存储能力，现有基于文件的设计严重依赖内核页缓存，导致缓存抖动、延迟不可预测且软件开销高。我们提出DUAL-BLADE，一种双路径KV驻留框架，根据运行时内存状况动态将KV张量分配至页缓存路径或直接NVMe路径。直接路径通过映射连续逻辑块地址（LBA）区域绕过文件系统，实现低开销存储访问。DUAL-BLADE还引入自适应流水线并行，使存储I/O与GPU DMA重叠，提升推理吞吐。评估表明，该方案显著缓解了I/O瓶颈，在不同内存预算下，预填充和解码延迟分别降低最多33.1%和42.4%，固态盘利用率提升2.2倍。
原文摘要 · Abstract (English)
The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。