用视觉自监督指令提升多模态模型的细粒度视觉推理能力
Boosting Visual Instruction Tuning with Self-Supervised Guidance
- 将旋转预测等自监督任务转为图文指令对，强制模型依赖视觉信息
- 仅用3-10%的新增指令，显著提升多个视觉任务表现
- 无需标注、架构修改或额外训练阶段，适配性强
多模态大模型在多数视觉语言任务中表现良好，但在需要细粒度视觉推理的任务上仍受限。研究表明，问题并非源于弱视觉表征，而是指令微调阶段对视觉信息利用不足，许多任务仅靠语言先验即可部分解决。本文提出一种轻量级方法，在视觉指令微调中引入少量以自然语言表述的视觉锚定自监督任务（如旋转预测、颜色匹配、跨视图对应）。通过将经典自监督预训练任务转化为图像-指令-回答三元组，加入仅依赖视觉证据才能解答的监督信号。该方法无需人工标注、不改变模型结构、不增加训练阶段。在多种模型、训练策略和基准测试中，仅注入3%-10%此类指令，即在视觉主导评估上实现一致性能提升。结果表明，通过调整训练数据分布，以视觉锚定的自监督任务增强指令微调，是提升多模态模型视觉推理能力的有效手段。代码开源：https://github.com/sirkosophia/V-GIFT
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual reasoning. Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets, we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning with visually grounded SSL tasks as a powerful lever for improving visual reasoning in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。