动态优化边缘LLM中LoRA适配器的缓存与路由,提升响应速度。
POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Serving
- 采用双时间尺度强化学习,联合优化适配器缓存和请求路由。
- 实验显示比静态策略快30%以上,且性能随适配器数量增长而提升。
- 适合部署在资源受限的边缘设备上运行大模型的场景。
边缘部署大型语言模型(LLMs)越来越依赖轻量级的LoRA适配器库,但GPU/DRAM只能驻留少量适配器。当请求需使用非驻留适配器时,必须从存储加载权重,带来可测量延迟。这形成一个双时间尺度在线控制问题:慢速决策选择哪些适配器保留在高速内存中,快速决策则将每个请求路由到上下文相关效用未知的适配器。两者紧密耦合:缓存决定探索成本,路由决定哪些适配器能获得有用反馈。本文将该联合缓存-路由问题建模为双时间尺度上下文博弈,提出POLAR(Paging and Online Learning for Adapter Routing)。POLAR结合缓存感知的LinUCB路由器与基于周期的缓存控制器。研究两种变体:固定周期版本在任意上下文中提供最坏情况后悔保证;倍增周期版本POLAR+引入强制探索与改进缓存优化,在随机规律性和可缓存条件下实现$ ilde{/mathcal{O}}(d oot{NT}+ oot{KT})$的次线性后悔,其中$N$为适配器数,$K$为缓存大小,$d$为上下文维度,$T$为时域。路由项达到标准上下文博弈速率,表明内存层次结构未根本拖慢学习。使用15个真实Qwen2.5-7B的LoRA适配器及实测GPU页错误延迟进行实验,结果表明自适应缓存控制显著优于非自适应基线,并呈现与理论一致的缩放趋势。
原文摘要 · Abstract (English)
Edge deployment of large language models (LLMs) increasingly relies on libraries of lightweight LoRA adapters, yet GPU/DRAM can keep only a small resident subset at a time. Serving a request through a non-resident adapter requires paging its weights from storage, incurring measurable latency. This creates a two-timescale online control problem: on a slow timescale, the system selects which adapters remain resident in fast memory, while on a fast timescale it routes each request to an adapter whose context-dependent utility is unknown a priori. The two decisions are tightly coupled: the cache determines the cost of exploration, and the router determines which adapters receive informative feedback. We formulate this joint caching-and-routing problem as a two-timescale contextual bandit and propose POLAR (Paging and Online Learning for Adapter Routing). POLAR pairs a cache-aware LinUCB router with an epoch-based cache controller. We study two variants. A fixed-epoch version provides a robust baseline with worst-case regret guarantees under arbitrary contexts. An epoch-doubling version, POLAR+, adds forced exploration and improved cache optimization to achieve $\widetilde{\mathcal{O}}(d\sqrt{NT}+\sqrt{KT})$ sublinear regret under stochastic regularity and cacheability conditions, where $N$ is the adapter count, $K$ the cache size, $d$ the context dimension, and $T$ the horizon. The routing term matches the standard contextual-bandit rate up to logarithmic factors, showing that the memory hierarchy does not fundamentally slow routing learning. Experiments using 15 real LoRA adapters for Qwen2.5-7B together with measured GPU paging latencies show that adaptive cache control substantially outperforms non-adaptive baselines and exhibits scaling trends consistent with the theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。