让多模态大模型自己发现视觉盲区并提升感知能力
Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots

- 通过对比反事实场景,自动识别模型未充分利用的视觉区域
- 在12个基准上超越6种自进化方法,最高提升3.60分
- 无需外部标注或强模型,适合想提升视觉理解的开发者
多模态大语言模型的自我改进通常依赖粗粒度的奖励信号。知识蒸馏可提供更丰富的逐标记级监督,但现有视觉蒸馏方法常依赖外部标注或更强模型构建特权上下文。本文提出首个完全自洽的密集、在策略、逐标记级视觉自蒸馏框架CVPD(对比反事实视觉过程蒸馏)。CVPD通过分析模型响应变化,识别出在局部放大时答案分布显著变化、移除后整体行为不变的视觉盲区,揭示模型能编码但未稳定利用的感知信息。提出三门控反事实准则,直接从模型自身输出中提取对比监督信号用于自蒸馏。在Qwen3-VL-8B-Instruct上,CVPD在12个基准上优于6种自进化基线,包括依赖GPT-4o外部监督的方法,无一退步;在OCRBench、MMStar细粒度感知与逻辑推理任务上分别取得+3.60、+3.38、+3.08的提升,同时保持或提升在更广泛多模态基准上的表现。
原文摘要 · Abstract (English)
Self-improvement for multimodal large language models (MLLMs) is typically driven by reward-based methods that provide only coarse scalar feedback. Distillation offers a richer alternative through dense token-level supervision, but in the visual domain it usually depends on privileged context constructed using external annotations and tools, or stronger models. We introduce \textbf{CVPD} (Contrastive Counterfactual Visual Process Distillation), which, to the best of our knowledge, is the first fully self-contained framework for dense, on-policy, token-level visual self-distillation for MLLMs. CVPD identifies visual blind spots where zooming into a region changes and sharpens the model's answer distribution, while removing the same region leaves the full-image behavior largely unchanged. Such regions reveal perceptual information that the model can encode but fails to consistently utilize under full-image conditioning. We propose a three-gate Counterfactual Criterion that identifies these regions directly from the model's own responses and converts them into dense contrastive supervision for self-distillation. On Qwen3-VL-8B-Instruct, CVPD outperforms six self-evolving baselines across twelve benchmarks, including methods that rely on external GPT-4o supervision, without a single regression. It achieves gains of $+3.60$ on OCRBench, $+3.38$ on MMStar Fine-Grained Perception, and $+3.08$ on MMStar Logical Reasoning, while maintaining or improving performance on broader multimodal benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。