让AI在图像问答中显式推理,提升准确率与可解释性。
StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering
- 用符号关系路径和自然语言解释构建双路径推理链
- 在OK-VQA上准确率最高提升11.3%,超越最强基线
- 无需外部知识库,仅用开源大模型即可实现结构化推理
基于知识的视觉问答(KVQA)要求模型在图像中定位实体并推理事实知识。近期工作提出了隐式知识变体IK-KVQA,其中多模态大语言模型(MLLM)是唯一知识来源,答案生成不依赖外部检索。然而现有IK-KVQA方法通常采用仅答案监督的训练方式:推理过程隐含、解释力弱且泛化能力脆弱。本文提出StaR-KVQA框架,引入双路径结构化推理轨迹——文本与视觉联合的符号关系路径,以及路径引导的自然语言解释——为模型提供强归纳偏置,优于通用答案监督。这些轨迹作为模态感知的支架,引导模型聚焦相关实体与属性,比通用思维链监督更具结构,又不限定固定推理路径。使用单一开源MLLM,StaR-KVQA构建并筛选轨迹,生成离线增强数据集,并进行结构感知自蒸馏;整个过程无需外部检索器、验证器或人工知识库,推理仅需一次自回归生成。在多个基准测试中,StaR-KVQA持续提升答案准确率与中间推理透明度,在OK-VQA上较最强基线最高提升+11.3%。
原文摘要 · Abstract (English)
Knowledge-based Visual Question Answering (KVQA) requires models to ground entities in images and reason over factual knowledge. Recent work has introduced its implicit-knowledge variant, IK-KVQA, where a multimodal large language model (MLLM) is the sole knowledge source and answers are produced without external retrieval. Existing IK-KVQA approaches, however, are typically trained with answer-only supervision: reasoning remains implicit, justifications are often weak or inconsistent, and generalization after standard supervised fine-tuning (SFT) can be brittle. We propose StaR-KVQA, a framework that equips IK-KVQA with dual-path structured reasoning traces - symbolic relation paths over text and vision together with path-grounded natural-language explanations - to provide a stronger inductive bias than generic answer-only supervision. These traces act as modality-aware scaffolds that guide the model toward relevant entities and attributes, offering more structure than generic chain-of-thought supervision while not constraining reasoning to any single fixed path. With a single open-source MLLM, StaR-KVQA constructs and selects traces to build an offline trace-enriched dataset and then performs structure-aware self-distillation; no external retrievers, verifiers, or curated knowledge bases are used, and inference is a single autoregressive pass. Across benchmarks, StaR-KVQA consistently improves both answer accuracy and the transparency of intermediate reasoning, achieving up to +11.3% higher answer accuracy on OK-VQA over the strongest baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。