用双奖励强化学习教会多模态大模型何时该思考、如何思考。
SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
- 设计双奖励机制，分别评估思考质量和是否需要深度推理。
- 在4B和8B模型上显著提升推理能力，效果接近GPT-4o。
- 有效减少幻觉，适合追求可靠推理的多模态应用开发者。
我们提出SAIL-RL，一种基于强化学习的后训练框架，通过教导多模态大语言模型（MLLMs）何时及如何思考，增强其推理能力。现有方法受限于仅以结果为导向的监督，无法保证推理质量；且采用统一思考策略，导致简单任务过度思考、复杂任务思考不足。SAIL-RL引入双奖励机制：思维奖励（Thinking Reward）通过事实依据、逻辑连贯性和答案一致性评估推理质量；判断奖励（Judging Reward）动态决定是否需深度推理或直接作答。在SAIL-VL2基准测试中，SAIL-RL在4B和8B规模下均显著提升推理与多模态理解性能，表现媲美GPT-4o等商用闭源模型，并大幅降低幻觉率，为构建更可靠、自适应的MLLM提供了原则性方案。代码将开源于https://github.com/BytedanceDouyinContent/SAIL-RL。
原文摘要 · Abstract (English)
We introduce SAIL-RL, a reinforcement learning (RL) post-training framework that enhances the reasoning capabilities of multimodal large language models (MLLMs) by teaching them when and how to think. Existing approaches are limited by outcome-only supervision, which rewards correct answers without ensuring sound reasoning, and by uniform thinking strategies, which often lead to overthinking on simple tasks and underthinking on complex ones. SAIL-RL addresses these challenges with a dual reward system: the Thinking Reward, which evaluates reasoning quality through factual grounding, logical coherence, and answer consistency, and the Judging Reward, which adaptively determines whether deep reasoning or direct answering is appropriate. Experiments on the state-of-the-art SAIL-VL2 show that SAIL-RL improves reasoning and multimodal understanding benchmarks at both 4B and 8B scales, achieving competitive performance against commercial closed-source models such as GPT-4o, and substantially reduces hallucinations, establishing it as a principled framework for building more reliable and adaptive MLLMs. The code will be available at https://github.com/BytedanceDouyinContent/SAIL-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。