让视觉语言模型学会识别图像描述中的细微错误,提升真实感知能力。
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
- 用人工注入的细微错位描述训练模型定位错误片段。
- 在多个视觉任务上显著提升表现,且泛化到抽象图像与视觉数学。
- 适合想提升模型真实理解力的研究者和开发者。
强化学习在大语言模型微调中表现出色,尤其适用于数学推理或代码生成等可验证任务。然而,将此成功扩展至视觉语言模型(VLMs)的视觉感知仍受制于缺乏兼具挑战性与明确可验证性的视觉任务。为此,我们提出ViCrit(视觉描述幻觉批评者),一种用于训练VLMs的强化学习代理任务:在200词的人类撰写图像描述中,注入一个微小的合成视觉错误——仅改变物体、属性、数量或空间关系的几个词——要求模型在给定图像和修改后描述的情况下,准确找出被污染的片段。该设计保持了完整的感知难度,同时提供二值、精确匹配的奖励信号,易于计算且无歧义。使用ViCrit任务训练的模型在多种视觉语言基准测试中均取得显著提升,且性能迁移至自然图像之外的抽象图像推理与视觉数学任务,展现出学习感知而非仅记忆视觉对象的潜力。为便于评估,我们进一步构建了类别平衡的诊断基准ViCrit-Bench,系统性地探测不同图像领域与错误类型下的感知偏差。结果表明,精细的幻觉批评是一种有效且可泛化的视觉感知增强目标。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has shown great effectiveness for fine-tuning large language models (LLMs) using tasks that are challenging yet easily verifiable, such as math reasoning or code generation. However, extending this success to visual perception in vision-language models (VLMs) has been impeded by the scarcity of vision-centric tasks that are simultaneously challenging and unambiguously verifiable. To this end, we introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions. Starting from a 200-word captions, we inject a single, subtle visual description error-altering a few words on objects, attributes, counts, or spatial relations-and task the model to pinpoint the corrupted span given the image and the modified caption. This formulation preserves the full perceptual difficulty while providing a binary, exact-match reward that is easy to compute and unambiguous. Models trained with the ViCrit Task exhibit substantial gains across a variety of VL benchmarks. Crucially, the improvements transfer beyond natural-image training data to abstract image reasoning and visual math, showing promises of learning to perceive rather than barely memorizing seen objects. To facilitate evaluation, we further introduce ViCrit-Bench, a category-balanced diagnostic benchmark that systematically probes perception errors across diverse image domains and error types. Together, our results demonstrate that fine-grained hallucination criticism is an effective and generalizable objective for enhancing visual perception in VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。