通过分離感知與推理,讓模型更依賴真實視覺證據而非捷徑。
Unbiased Visual Reasoning with Controlled Visual Inputs
- 用固定視覺模型只回答客觀問題,由文本大模型負責規劃與推理。
- 在SpuriVerse上提升16.29%(Qwen)和6.77%(Llama)的魯棒性。
- 適合需要可靠、可解釋視覺推理的應用場景。
端到端視覺-語言模型常依賴偽相關而非因果視覺證據回答問題,微調後更易走捷徑。我們提出VISTA(基於文本分析的視覺信息分離),透過顯式信息瓶頸將感知與推理分離:固定視覺模型僅回應短暫客觀感知查詢,而純文本大模型則分解問題、規劃查詢並以自然語言整合視覺事實。此受控介面構建獎勵對齊環境,用強化學習訓練無偏視覺推理。以Qwen2.5-VL與Llama3.2-Vision為傳感器,僅用641個精心挑選的多步問題訓練GRPO,VISTA在SpuriVerse上分別提升16.29%(Qwen-2.5-VL-7B)與6.77%(Llama-3.2-Vision-11B)的魯棒性,同時在MMVP與平衡版SeedBench上保持競爭力。VISTA能跨未見傳感器泛化,並識別與恢復傳感器失敗。人工分析顯示,VISTA的推理過程更中立、少依賴偽特徵,且更明確基於視覺證據。
原文摘要 · Abstract (English)
End-to-end Vision-language Models (VLMs) often answer visual questions by exploiting spurious correlations instead of causal visual evidence, and can become more shortcut-prone when fine-tuned. We introduce VISTA (Visual-Information Separation for Text-based Analysis), a modular framework that decouples perception from reasoning via an explicit information bottleneck. A frozen VLM sensor is restricted to short, objective perception queries, while a text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language. This controlled interface defines a reward-aligned environment for training unbiased visual reasoning with reinforcement learning. Instantiated with Qwen2.5-VL and Llama3.2-Vision sensors, and trained with GRPO from only 641 curated multi-step questions, VISTA significantly improves robustness to real-world spurious correlations on SpuriVerse (+16.29% with Qwen-2.5-VL-7B and +6.77% with Llama-3.2-Vision-11B), while remaining competitive on MMVP and a balanced SeedBench subset. VISTA transfers robustly across unseen VLM sensors and is able to recognize and recover from VLM perception failures. Human analysis further shows that VISTA's reasoning traces are more neutral, less reliant on spurious attributes, and more explicitly grounded in visual evidence than end-to-end VLM baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。