让多模态模型输出更适配大模型推理,实现无需重训的性能提升。
Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
- 将感知与推理分离,用外部大语言模型做推理
- 通过强化学习优化视觉描述,使输出更符合任务需求
- 可搭配任意先进文本推理模型,推理时灵活升级
近期推理型语言模型在文本推理上取得显著进展,而多模态大语言模型(MLLMs)仍因内部语言模型陈旧而滞后。升级这些模型通常代价高昂,需重新进行耗时的视觉-语言对齐训练。为此,我们提出感知-推理解耦机制,将MLLM的推理模块模块化,使其可被替换。该方法重新定义MLLM角色:将多模态输入转化为详尽的文本输出,供任何强大的外部纯文本推理模型处理。为对齐感知输出与最终推理任务,我们提出一种新型强化学习算法——视觉感知优化(VPO),根据外部推理模型生成答案的正确性奖励MLLM,从而生成忠实且与查询相关的描述。结合该解耦流程与VPO,我们构建了推理对齐感知解耦(RAPID)方法。实验表明,RAPID在多模态推理基准测试中取得显著性能提升。关键的是,一旦经由VPO训练,该MLLM可与任意前沿语言模型推理器搭配,在推理阶段实现一致性能改进,无需重新训练。
原文摘要 · Abstract (English)
Recent breakthroughs in reasoning language models have significantly advanced text-based reasoning. On the other hand, Multi-modal Large Language Models (MLLMs) still lag behind, hindered by their outdated internal LLMs. Upgrading these LLMs is often prohibitively expensive, as it requires costly vision-language alignment retraining. To address this issue, we introduce Perception-Reasoning Decoupling, which modularizes the MLLM's reasoning component and makes it easily replaceable. This approach redefines the MLLM's role to convert multi-modal inputs into detailed textual outputs that can be processed by any powerful, external, text-only LLM reasoners. To align the MLLM's perceptual output with the final reasoning task, we propose a novel reinforcement learning algorithm called Visual Perception Optimization (VPO). VPO rewards the MLLM based on the correctness of answers generated by the external reasoner to produce faithful and query-relevant captions. Together, this decoupling pipeline and VPO form our Reasoning-Aligned PerceptIon Decoupling (RAPID) approach. Empirical results show that RAPID achieves significant performance gains on multi-modal reasoning benchmarks. Crucially, RAPID enables a novel inference-time scaling paradigm: Once trained with VPO, the MLLM can be paired with any state-of-the-art LLM reasoner for consistent performance improvement without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。