让大模型学多模态时保住语言能力，只动该动的神经元。
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
- 按神经元敏感度分配更新强度，关键语言神经元受保护。
- 在11个语言基准上恢复94.5%的语言能力退化损失。
- 轻量无架构依赖，适合各类大模型的多模态扩展。
大型语言模型（LLMs）的多模态扩展带来了新的感知能力，但常导致预训练阶段积累的语言智能退化。本文从内部适应动态角度研究此现象，发现预训练LLM中的神经元在多模态学习中表现出异质性可塑性：部分神经元对语言能力至关重要，而另一些则更易适应多模态知识。基于此，我们提出NeuPAT（Neuron-aware Plasticity Allocation Tuning），一种轻量且与架构无关的框架，在多模态指令微调过程中为神经元分配细粒度更新约束。NeuPAT通过小规模探测阶段估计神经元适应模式，选择性保护语言敏感神经元，同时通过更具可塑性的神经元促进多模态适应。在多种LLM家族上的实验表明，NeuPAT在11个语言基准上恢复了94.5%由常规微调造成的语言能力退化，同时保持相当的多模态性能，为能力保全的多模态扩展提供有效方案。
原文摘要 · Abstract (English)
Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。