arXiv:2608.16477cs.LG2026-08

提前迁移大模型推理缓存,显著缩短用户切换基站时的等待时间。

Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN

论文配图:Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN
图 1 · 摘自论文原文
  • 在用户切换前预估目标基站,提前准备推理状态
  • 平均服务中断时间减少2.28至89.68倍,跨站延迟降低16%~50%
  • 适合移动边缘计算中对实时性要求高的大模型应用

AI-RAN将大语言模型服务贴近移动用户,但用户切换基站时可能造成推理状态与设备分离:用户连接到目标基站(gNB),而不断增长的关键-值(KV)缓存仍留在源端。若保持源端推理,虽能保证服务连续性,却持续增加跨令牌延迟(ITL);若在目标端恢复状态,则需缓存传输、重新计算或两者结合,仅在切换后执行,直接延长服务中断时间(SIT)。本文提出Pallas,一种主动式KV缓存迁移框架,在切换前即于预测的目标端预先准备推理状态,同时源端继续推理与分发令牌。触发准备时,Pallas将令牌序列分为稳定的历史前缀与动态的后缀。目标端通过本地预填充重建前缀,源端则流式发送后缀生成的KV块。切换发生时,目标端拼接两部分形成最新缓存并本地恢复解码,仅未完成的准备工作影响SIT。在线调度器根据移动性预测与运行时数据选择预取窗口,决定准备开始的时机。基于vLLM原型,在三个LLM及100–500 Mbps gNB间链路下,相比目标端恢复方法,平均SIT降低2.28–89.68倍;相比源端转发,平均ITL降低16.0%–50.0%。

原文摘要 · Abstract (English)

AI-RAN brings large language model (LLM) serving close to mobile users, but cellular handover can separate an active request from its inference state: the user attaches to a target base station (gNB) while the large and growing key-value (KV) cache remains at the source. Retaining inference at the source preserves service continuity but persistently increases inter-token latency (ITL), whereas recovering the state at the target restores serving locality but requires KV-cache transfer, recomputation, or a combination of both only after handover, directly prolonging service interruption time (SIT). This work presents Pallas, a \textit{proactive} KV-cache migration framework that prepares the inference state at the predicted target before handover, in parallel with ongoing source-side inference and token delivery. At the preparation trigger, Pallas partitions the token sequence into a stable historical prefix and an evolving suffix. The target reconstructs the prefix through local prefill, while the source streams the KV blocks generated for the suffix. At handover, the target assembles both portions into an up-to-date KV cache and resumes decoding locally, leaving only unfinished preparation to contribute to SIT. An online scheduler selects the \textit{prefetching window}, which determines how early preparation begins before handover, based on mobility predictions and runtime telemetry. Across three LLMs and $100$--$500~\mathrm{Mbps}$ inter-gNB links, our vLLM-based prototype reduces average SIT by factors of $2.28$--$89.68$ over target-side recovery approaches and lowers average ITL by $16.0\%$--$50.0\%$ compared with source-side forwarding.

大模型推理边缘计算缓存迁移AI-RAN

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。