用扩散模型提升视觉语言模型，仅需少量数据一天完成训练
Diffusion Instruction Tuning
- 对齐视觉语言模型与扩散模型的注意力机制，无需额外编码器
- 仅用13万样本，实现30%性能提升，医疗问答任务提升68%
- 适合想高效优化多模态大模型的研究者和开发者
我们提出Lavender，一种简单的监督微调方法，通过利用Stable Diffusion等先进图像生成模型，显著提升视觉语言模型（VLM）性能。Lavender在微调过程中对齐VLM Transformer中的文本-视觉注意力与Stable Diffusion所用的对应注意力，而非分别适配独立编码器。该对齐增强了模型的视觉理解能力，在分布内和分布外任务中均表现优异。Lavender仅需0.13百万（13万）训练样本，仅为典型大规模SFT数据集的2.5%，可在标准硬件（8张GPU）上单日完成微调。它持续改进当前开源多模态LLM（如Llama-3.2-11B、MiniCPM-Llama3-v2.5），在任务上实现最高30%的性能提升，并在具有挑战性的分布外医学QA任务中达到68%的显著增益。通过以极少监督高效迁移图像生成器的视觉专长，Lavender为更精确的视觉语言系统提供了可扩展的解决方案。所有代码、训练数据和模型将公开于https://astrazeneca.github.io/vlm/
原文摘要 · Abstract (English)
We introduce Lavender, a simple supervised fine-tuning (SFT) method that boosts the performance of advanced vision-language models (VLMs) by leveraging state-of-the-art image generation models such as Stable Diffusion. Specifically, Lavender aligns the text-vision attention in the VLM transformer with the equivalent used by Stable Diffusion during SFT, instead of adapting separate encoders. This alignment enriches the model's visual understanding and significantly boosts performance across in- and out-of-distribution tasks. Lavender requires just 0.13 million training examples, 2.5% of typical large-scale SFT datasets, and fine-tunes on standard hardware (8 GPUs) in a single day. It consistently improves state-of-the-art open-source multimodal LLMs (e.g., Llama-3.2-11B, MiniCPM-Llama3-v2.5), achieving up to 30% gains and a 68% boost on challenging out-of-distribution medical QA tasks. By efficiently transferring the visual expertise of image generators with minimal supervision, Lavender offers a scalable solution for more accurate vision-language systems. All code, training data, and models will be shared at https://astrazeneca.github.io/vlm/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。