让视觉指令模型更懂图，提升理解力并减少幻觉
Learning to Instruct for Visual Instruction Tuning
- 在指令和回复序列中同时加入损失函数，增强视觉理解
- 在多个基准上提升最高达9%的性能，图文生成提升18%
- 无需额外数据，计算开销极小，适合实际部署
我们提出 L2T，一种视觉指令调优（VIT）的改进方法。尽管 VIT 使多模态大模型（MLLMs）具备出色的多模态能力，但现有设计常导致过拟合和捷径学习，影响性能。其根源在于过度强调指令遵循，忽视了对视觉信息的主动理解。受此启发，L2T 采用简单有效的方法：将损失函数同时引入指令和回复序列，无缝扩展训练数据，并抑制 MLLMs 对语言先验的过度依赖。基于此，L2T 在综合性多模态基准上实现最高达 9% 的相对性能提升，无需额外训练数据且计算开销可忽略。令人惊讶的是，L2T 显著提升了基础视觉能力，图文生成性能最高提升 18%，同时有效缓解了 MLLMs 的幻觉问题。代码已开源：https://github.com/Feng-Hong/L2T。
原文摘要 · Abstract (English)
We propose L2T, an advancement of visual instruction tuning (VIT). While VIT equips Multimodal LLMs (MLLMs) with promising multimodal capabilities, the current design choices for VIT often result in overfitting and shortcut learning, potentially degrading performance. This gap arises from an overemphasis on instruction-following abilities, while neglecting the proactive understanding of visual information. Inspired by this, L2T adopts a simple yet effective approach by incorporating the loss function into both the instruction and response sequences. It seamlessly expands the training data, and regularizes the MLLMs from overly relying on language priors. Based on this merit, L2T achieves a significant relative improvement of up to 9% on comprehensive multimodal benchmarks, requiring no additional training data and incurring negligible computational overhead. Surprisingly, L2T attains exceptional fundamental visual capabilities, yielding up to an 18% improvement in captioning performance, while simultaneously alleviating hallucination in MLLMs. Github code: https://github.com/Feng-Hong/L2T.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。