无需额外训练,用激活值直接路由LoRA参数记忆库。
Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory

- 将零样本路由重构为解码外部参数记忆的激活信号。
- 单次预填充即可评分所有LoRA,性能超越现有方法。
- 适合追求轻量级部署的模型优化研究者。
随着参数化记忆的发展,基于LoRA的外部参数化记忆(EPM)成为模块化解决方案,但现有路由方法常引入额外训练、部署与维护开销。这引发一个自然问题:能否在不维护额外路由组件的前提下实现LoRA-EPM的路由?然而,现有零样本LoRA路由方法在EPM场景下仍存在两大缺陷:(1) 评估分散于不同任务设置,未围绕EPM访问组织;(2) 路由信号缺乏统一视角以指导系统性改进。为此,我们构建了PMD-Bench,涵盖文档级、领域级知识与任务技能三类任务,并提出参数化记忆解码(PMD),首个将零样本LoRA路由系统性提升的框架,将其重构为对外部参数化记忆的激活解码。基于PMD,我们进一步提出PMDRouter,通过单次基础模型预填充中的响应幅度对每个LoRA进行评分。在PMD-Bench上的实验表明,PMDRouter在多个零样本路由设置中均取得最强内部信号表现。结果验证了零样本路由的可行性,并表明PMD可作为通用框架用于改进零样本路由方法。
原文摘要 · Abstract (English)
With the rise of parametric memory, LoRA-based External Parametric Memory (EPM) has emerged as a modular solution, but existing routing methods often introduce additional training, deployment, and maintenance overhead. This raises a natural question: can a LoRA-based EPM bank be routed without maintaining an additional routing component? However, existing zero-shot LoRA routing methods still face two problems under the EPM setting: (1) their evaluations are scattered across different task settings rather than organized around EPM access, and (2) their routing signals lack a unified perspective to guide systematic improvement. To address these problems, we organize PMD-Bench, covering document-level, domain-level knowledge, and task-skill, and propose Parametric Memory Decoding (PMD), the first framework designed to systematically improve zero-shot LoRA routing by reframing it as decoding activations over external parametric memory. Based on PMD, we further instantiate PMDRouter, which scores each LoRA by its response magnitude from a single base-model prefill. Experiments on PMD-Bench show that PMDRouter achieves the strongest internal-signal performance across multiple zero-shot routing settings. These results demonstrate the feasibility of zero-shot LoRA routing and suggest that PMD can serve as a general framework for improving zero-shot routing methods. Sources: Github (https://anonymous.4open.science/r/Parametric-Memory-Decoding-872A/)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。