自动构建多步视觉思维链奖励模型，提升大模型推理准确性。
Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program
- 用代码生成与分析替代人工标注，实现步骤级多维思维链训练。
- 在多个基准上显著降低幻觉率，推理能力提升，跨训练/推理缩放更优。
- 适合需要高可靠推理的多模态大模型研究者与开发者。
大语言模型中奖励信号的使用进展迅速，但向多模态领域迁移时面临标注成本高、依赖单步奖励、评估不足等挑战。为此，我们提出SVIP，一种自动训练步骤级多维度思维链（CoT）奖励模型的新方法。该方法通过生成解决视觉任务的代码，并将代码块分析转化为思维链步骤的评估样本。采用名为TriAtt-CoT的多头注意力机制训练SVIP-Reward模型。实验表明，SVIP-Reward在多模态大模型全过程中表现优异，在训练与推理缩放下均提升性能，且在多个基准测试中表现更佳，同时减少幻觉并增强推理能力。
原文摘要 · Abstract (English)
Recent advancements in reward signal usage for Large Language Models (LLMs) are remarkable. However, significant challenges exist when transitioning reward signal to the multimodal domain, including labor-intensive annotations, over-reliance on one-step rewards, and inadequate evaluation. To address these issues, we propose SVIP, a novel approach to train a step-level multi-dimensional Chain-of-Thought~(CoT) reward model automatically. It generates code for solving visual tasks and transforms the analysis of code blocks into the evaluation of CoT step as training samples. Then, we train SVIP-Reward model using a multi-head attention mechanism called TriAtt-CoT. The advantages of SVIP-Reward are evident throughout the entire process of MLLM. We also introduce a benchmark for CoT reward model training and testing. Experimental results demonstrate that SVIP-Reward improves MLLM performance across training and inference-time scaling, yielding better results on benchmarks while reducing hallucinations and enhancing reasoning ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。