让手机运行长文本生成更准更快,动态调整缓存管理。
DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones
- 动态自适应缓存分区,避免频繁数据迁移。
- 提升检索准确率1.38倍,加速1.47倍,延迟更低。
- 适合移动端长文本生成,也适用于多层存储系统。
随着对类人推理、多轮对话和长文本回复需求的增长,大语言模型(LLMs)需支持高效长序列解码。然而,受手机内存容量限制,关键值缓存(KVCache)的内存开销随序列长度线性增加,成为瓶颈。现有基于检索的方法将KVCache卸载至闪存并用聚类索引检索相关条目,但随着解码进行,缓存分布变化导致静态或局部聚类更新逐渐失准,造成关键信息遗漏或冗余读取。这一问题在手机特有的带宽、IOPS和内存容量限制下更为严重。本文提出DynaKV,首个针对手机端长序列解码的自适应缓存管理方案,融合三项核心技术:(1) 无迁移聚类自适应,解码中动态分裂聚类而无需额外传输;(2) 以连续性为中心的闪存管理,关联相关条目与聚类,采用双头布局优化更新效率;(3) 高效内存缓存设计,虚拟化跨DRAM与闪存的缓存空间,并扩展替换策略以匹配聚类级访问模式。评估表明,相比最先进方案,DynaKV在准确率上平均提升1.38倍,端到端延迟降低1.47倍。其核心洞察可自然推广至其他长上下文任务及多级存储架构。
原文摘要 · Abstract (English)
As the demand for human-like reasoning, multi-turn dialogues, and long-form responses grows, large language models (LLMs) are increasingly expected to support efficient and effective long-sequence decoding. However, due to limited DRAM capacity, long-seuqence LLM decoding on smartphones is constrained by the key-value cache (KVCache), whose memory footprint increases linearly with sequence length. Retrieval-based methods mitigate DRAM pressure by offloading KVCache to flash and retrieving query-relevant entries through cluster-based indexing. Unfortunately, as decoding progresses, KVCache distribution shifts render static or local cluster updates progressively misaligned, excluding essential entries or fetching redundant ones. These issues are further exacerbated by smartphone-specific limitations in bandwidth, IOPS, and memory capacity. We propose DynaKV, the first adaptive KVCache management approach that jointly addresses accuracy and efficiency for long-sequence decoding on smartphones. DynaKV integrates three key techniques: (1) Migration-Free Cluster Adaptation, which adaptively splits clusters during retrieval without incurring additional transfers; (2) Continuity-Centric Flash Management, which co-locates correlated entries and clusters and employs a dual-head layout for efficient updates; and (3) Memory-Efficient Cache Design, which virtualizes cache space across DRAM and flash and extends replacement policies to align with cluster-level access patterns. Evaluations demonstrate that DynaKV improves retrieval accuracy and reduces end-to-end latency compared to state-of-the-art solutions, achieving average gains of $1.38\times$ in accuracy and $1.47\times$ speedups. Furthermore, the insights of DynaKV naturally extend to other long-context workloads and multi-tier memory hierarchies, underscoring its broader applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。