arXiv:2608.22643cs.DCcs.LG2026-08中稿 · the 55th Internati…

通过预测稀疏激活差异,实现边缘设备上大模型的高效存储预取。

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

  • 基于神经网络激活的时序局部性,仅预取新增权重行。
  • 在受限内存下实现7.9至12.0倍推理加速。
  • 适合内存不足的大模型边缘部署场景。

将大语言模型部署在边缘设备上日益受到模型规模与可用内存之间差距的制约。现有方法如量化、小型化模型和卸载虽可提升有效内存上限,但仍假设模型可压缩或分片以适应预算。本文针对更困难的‘模型超出内存’场景,即模型始终大于驻留内存,存储成为关键路径上的活跃数据源。我们发现自回归解码中MLP的活跃神经元具有强时序局部性:约82%-85%的活跃神经元在前后词元间保持不变。这意味着当前词元所需大部分稀疏权重已在内存中,只需从存储加载新需行。提出NeuroPrefetcher,一种基于存储的稀疏推理系统,通过预测性差分预取利用此特性。在第0层后,仅需2.86%基础模型参数的单个GPU驻留预测器,在一次前向传播中预测所有下游MLP层的稀疏活动。运行时将预测与驻留缓冲区比对,仅对新增行调度应用级NVMe读取,取代反应式操作系统需求分页,实现显式、模型感知的权重迁移。在真实统一内存边缘硬件上,相比llama.cpp,在受限内存预算下获得7.9-12.0倍加速。

原文摘要 · Abstract (English)

Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.

边缘计算稀疏推理预取优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。