MoLKV让大模型专家选择更懂上下文，适合手机等设备运行。
Mixture of Lookup Key-Value Experts
- 每个专家存为键值对，输入通过查询匹配上下文相关专家。
- 小规模实验显示验证损失显著低于MoLE，提升推理效果。
- 适合资源受限设备，兼顾性能与存储效率，适合移动端部署。
近期研究提出了适用于终端设备推理的大语言模型架构，如查找专家混合模型（Mixture of Lookup Experts, MoLE）。MoLE的关键特性是每个词元ID对应一组专属专家，推理时仅激活输入词元对应的专家，因激活专家数量少，加载到内存的通信开销可忽略，从而可将专家参数离线存储，适配资源受限设备。然而，MoLE基于输入词元ID的上下文无关专家选择机制可能限制模型性能。为此，本文提出 extbf{M}ixture extbf{o}f extbf{L}ookup extbf{K}ey- extbf{V}alue Experts ( extbf{MoLKV}) 模型。在MoLKV中，每个专家以键值对形式组织。给定输入时，由输入生成的查询与当前序列缓存的键值专家交互，生成具有上下文感知能力的专家输出。该机制缓解了MoLE的局限性，实验结果表明，MoLKV在小规模评估中实现了显著更低的验证损失。
原文摘要 · Abstract (English)
Recent research has developed several LLM architectures suitable for inference on end-user devices, such as the Mixture of Lookup Experts (MoLE)~\parencite{jie_mixture_2025}. A key feature of MoLE is that each token id is associated with a dedicated group of experts. For a given input, only the experts corresponding to the input token id will be activated. Since the communication overhead of loading this small number of activated experts into RAM during inference is negligible, expert parameters can be offloaded to storage, making MoLE suitable for resource-constrained devices. However, MoLE's context-independent expert selection mechanism, based solely on input ids, may limit model performance. To address this, we propose the \textbf{M}ixture \textbf{o}f \textbf{L}ookup \textbf{K}ey-\textbf{V}alue Experts (\textbf{MoLKV}) model. In MoLKV, each expert is structured as a key-value pair. For a given input, the input-derived query interacts with the cached key-value experts from the current sequence, generating a context-aware expert output. This context-aware mechanism alleviates the limitation of MoLE, and experimental results demonstrate that MoLKV achieves significantly lower validation loss in small-scale evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。