让视觉问答更懂多语言，通过分步推理提升准确率
LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA
- 设计多阶段语言感知推理流程，融合文本、位置与语义信息
- 在多个数据集上比同类模型高9.5%，超越两倍大的商用模型
- 适合需要多语言视觉理解的工业场景，如跨语言客服系统
随着大规模视觉语言模型的发展，多语言视觉问答（mVQA）能力显著提升。链式思维（CoT）已被证明可增强可解释性与复杂推理能力。然而，现有方法主要依赖文本型CoT，对多语言多模态推理支持有限，限制了其在真实场景的应用。为此，我们提出首个语言感知视觉CoT框架LaV-CoT，结合多阶段推理流程：包含边界框的文本摘要、语言识别、空间物体级描述与逐步逻辑推理。基于该流程，设计自动化数据构建方法，通过迭代生成、修正与优化生成高质量多语言CoT标注数据。为提升推理与泛化能力，采用监督微调与语言感知组相对策略优化（GRPO）的两阶段训练范式，由可验证的多维度奖励（语言一致性、结构准确性、语义对齐）指导。在MMMB、Multilingual MMBench和MTVQA等公开数据集上的大量实验表明，LaV-CoT相比同规模开源基线最高提升~9.5%准确率，甚至超过规模大2倍的模型约2.6%。此外，其性能优于GPT-4o-0513与Gemini-2.5-flash等先进商用模型。我们还进行了线上A/B测试，在真实数据上验证了方法的有效性，具备工业部署潜力。代码已开源：https://github.com/HJNVR/LaV-CoT
原文摘要 · Abstract (English)
As large vision language models (VLMs) advance, their capabilities in multilingual visual question answering (mVQA) have significantly improved. Chain-of-thought (CoT) reasoning has been proven to enhance interpretability and complex reasoning. However, most existing approaches rely primarily on textual CoT and provide limited support for multilingual multimodal reasoning, constraining their deployment in real-world applications. To address this gap, we introduce LaV-CoT, the first Language-aware Visual CoT framework with Multi-Aspect Reward Optimization. LaV-CoT incorporates an interpretable multi-stage reasoning pipeline consisting of Text Summary with Bounding Box (BBox), Language Identification, Spatial Object-level Captioning, and Step-by-step Logical Reasoning. Following this reasoning pipeline, we design an automated data curation method that generates multilingual CoT annotations through iterative generation, correction, and refinement, enabling scalable and high-quality training data. To improve reasoning and generalization, LaV-CoT adopts a two-stage training paradigm combining Supervised Fine-Tuning (SFT) with Language-aware Group Relative Policy Optimization (GRPO), guided by verifiable multi-aspect rewards including language consistency, structural accuracy, and semantic alignment. Extensive evaluations on public datasets including MMMB, Multilingual MMBench, and MTVQA show that LaV-CoT achieves up to ~9.5% accuracy improvements over open-source baselines of similar size and even surpasses models with 2$\times$ larger scales by ~2.6%. Moreover, LaV-CoT outperforms advanced proprietary models such as GPT-4o-0513 and Gemini-2.5-flash. We further conducted an online A/B test to validate our method on real-world data, highlighting its effectiveness for industrial deployment. Our code is available at this link: https://github.com/HJNVR/LaV-CoT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。