让视觉语言模型通过自我反思改进推理能力
Vision-Language Models Can Self-Improve Reasoning via Reflection
- 通过自我精炼和筛选损失，迭代优化推理过程
- 在多个任务上提升23%至60%的推理准确率
- 适合需要强推理能力的多模态AI研究者
链式思维（CoT）已被证明能提升大语言模型的推理能力。然而，由于多模态场景复杂且高质量CoT数据难以获取，多模态大模型的CoT推理尚未得到充分研究。为此，我们提出一个简单有效的自训练框架R3V，通过反思链式推理过程，持续增强视觉-语言模型的推理能力。该框架包含两个交替进行的部分：（1）迭代构建正负样本推理数据集；（2）对推理过程进行反思以从错误中学习。具体地，引入自精炼与自选择损失，使模型能对比推理候选，修正错误推理并得出正确答案。在多种视觉语言任务上的实验表明，R3V显著提升了多模态大模型的推理性能，相对于GPT蒸馏基线实现了23%至60%的相对提升。此外，该方法支持生成解题后的自我反思，进一步通过测试时计算提升性能。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R3V, which iteratively enhances the model's Vision-language Reasoning by Reflecting on CoT Rationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R3V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23 to 60 percent over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。