无需预训练，低成本将视觉语言模型转为高效注意力结构。
MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
- 通过模态自适应掩码和独立压缩图文键值，实现轻量级转换。
- 在三个模型上复现原性能，键值缓存减少超70%。
- 适合追求推理加速与显存优化的部署场景。
随着视觉语言模型（VLMs）处理日益复杂的多模态任务，键值（KV）缓存的快速增长带来了严重的内存与计算瓶颈。尽管多头潜在注意力（MLA）能有效压缩KV缓存并加速推理，但如何在不进行昂贵预训练的前提下将现有VLM适配至MLA架构仍缺乏研究。本文提出MHA2MLA-VLM，一种参数高效且具备多模态感知能力的框架，可将现成VLM转换为MLA结构。核心包括：(1) 模态自适应部分旋转位置编码（partial-RoPE）策略，支持传统与多模态场景；(2) 模态解耦的低秩近似方法，分别压缩视觉与文本的键值空间。此外，采用参数高效的微调策略，并发现最小化输出激活误差比最小化参数距离更能显著降低性能损失。在三个代表性VLM上的实验表明，该方法仅需少量监督数据即可恢复原始性能，大幅减少KV缓存占用，并与量化技术无缝集成。
原文摘要 · Abstract (English)
As vision-language models (VLMs) tackle increasingly complex and multimodal tasks, the rapid growth of Key-Value (KV) cache imposes significant memory and computational bottlenecks during inference. While Multi-Head Latent Attention (MLA) offers an effective means to compress the KV cache and accelerate inference, adapting existing VLMs to the MLA architecture without costly pretraining remains largely unexplored. In this work, we present MHA2MLA-VLM, a parameter-efficient and multimodal-aware framework for converting off-the-shelf VLMs to MLA. Our approach features two core techniques: (1) a modality-adaptive partial-RoPE strategy that supports both traditional and multimodal settings by selectively masking nonessential dimensions, and (2) a modality-decoupled low-rank approximation method that independently compresses the visual and textual KV spaces. Furthermore, we introduce parameter-efficient fine-tuning to minimize adaptation cost and demonstrate that minimizing output activation error, rather than parameter distance, substantially reduces performance loss. Extensive experiments on three representative VLMs show that MHA2MLA-VLM restores original model performance with minimal supervised data, significantly reduces KV cache footprint, and integrates seamlessly with KV quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。