让视觉令牌直接学字,提升模型对细节的理解能力
DV-SFT: Direct Vision Supervision for Fine-Grained Visual Understanding

- 直接用图像中的文字给视觉令牌打标签,实现显式监督
- 在7个基准上均优于传统微调,细粒度理解明显提升
- 无需改模型结构,适合想快速提升视觉理解的开发者
多模态大语言模型通常仅对文本令牌进行端到端训练,视觉令牌作为视觉信息的核心载体,仅通过上下文隐式优化,导致视觉理解粗略。已有方法尝试监督视觉输入,但需额外解码器或前向传播,实用性受限。本文提出直接视觉监督微调(DV-SFT),在OCR场景中利用图像块与对应文字的直接对应关系,自动为每个视觉令牌标注其对应文字,并通过与文本相同的下一个词预测目标进行训练。该方法将MLLM视为黑箱,无需架构修改或额外前向传播。大量实验表明,直接视觉监督显著优于标准SFT,在三个域内和四个域外基准上均表现更优。分析显示,该方法有效增强细粒度视觉理解,提升多模态对齐效率。
原文摘要 · Abstract (English)
Multimodal large language models are typically trained end-to-end to predict ground-truth answers, yet supervision signals are applied exclusively to text tokens. Visual tokens, the core carriers of visual information, are optimized only implicitly as part of the context, leading to coarse-grained visual understanding. Prior works attempt to supervise visual inputs but inevitably rely on auxiliary components such as additional decoders or forward passes, because visual tokens lack readily interpretable labels. This limits their practical applicability. In this work, we propose \textbf{D}irect \textbf{V}ision \textbf{S}upervised \textbf{F}ine-\textbf{T}uning (DV-SFT), which constructs explicit, token-level supervision for visual tokens and trains them through the same next-token prediction objective used for text. Specifically, we exploit the direct vision--text correspondence in OCR-related scenarios and automatically label each visual token with the word in its corresponding image patch. DV-SFT treats the MLLM as a black box, requiring no architectural modifications or additional forward passes. Extensive experiments demonstrate the superiority of direct vision supervision. DV-SFT consistently outperforms standard SFT across three in-domain and four out-of-domain benchmarks. Further analyses show that vision supervision effectively enhances fine-grained visual understanding and achieves higher multimodal alignment efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。