通过注入定制噪声提升多模态模型性能,仅用1%-2%参数就超越全量微调。
Explore How to Inject Beneficial Noise in MLLMs
- 从变分推断视角重构推理过程,动态生成任务适配的跨模态噪声。
- 在QwenVL和LLaVA上超越全量微调,仅需调整1%~2%额外参数。
- 适合追求高效微调且关注跨模态对齐的多模态研究者。
多模态大语言模型(MLLMs)在多模态智能中日益重要,但现有微调方法常忽略跨模态异质性,限制其潜力。本文提出一种新微调策略:通过注入有益随机噪声,显著优于以往方法,甚至超越全量微调,且仅增加约1%~2%额外参数。提出的多模态噪声生成器(MuNG)通过冻结的MLLM注入定制噪声,实现高效模态微调。具体而言,我们从变分推断角度重构MLLM推理过程,设计出能动态分析图像-文本对跨模态关系的噪声生成器,生成任务自适应的有益噪声。该噪声有效抑制无关语义成分,显著提升跨模态表示对齐,并增强下游任务性能。在主流模型QwenVL和LLaVA上的实验表明,该方法超越全参数微调及其他现有微调方法,仅需调整约1%~2%额外参数。相关代码已上传至附录。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have played an increasingly important role in multimodal intelligence. However, the existing fine-tuning methods often ignore cross-modal heterogeneity, limiting their full potential. In this work, we propose a novel fine-tuning strategy by injecting beneficial random noise, which outperforms previous methods and even surpasses full fine-tuning, with minimal additional parameters. The proposed Multimodal Noise Generator (MuNG) enables efficient modality fine-tuning by injecting customized noise into the frozen MLLMs. Specifically, we reformulate the reasoning process of MLLMs from a variational inference perspective, upon which we design a multimodal noise generator that dynamically analyzes cross-modal relationships in image-text pairs to generate task-adaptive beneficial noise. Injecting this type of noise into the MLLMs effectively suppresses irrelevant semantic components, leading to significantly improved cross-modal representation alignment and enhanced performance on downstream tasks. Experiments on two mainstream MLLMs, QwenVL and LLaVA, demonstrate that our method surpasses full-parameter fine-tuning and other existing fine-tuning approaches, while requiring adjustments to only about $1\sim2\%$ additional parameters. The relevant code is uploaded in the supplementary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。