小模型推理能力迁移,大模型无需训练就能变聪明。
ProxyThinker: Test-Time Guidance through Small Visual Reasoners
- 用小模型输出减去大模型输出,改变解码过程
- 在空间、数学等难题上性能提升显著
- 速度快38倍,适合实际部署的推理加速
近期基于可验证奖励的强化学习进展推动了大视觉语言模型(LVLM)的视觉推理能力。然而,通过强化微调(RFT)训练LVLM计算成本高昂,限制了模型规模扩展。本文提出ProxyThinker,一种推理时技术,使大模型无需训练即可继承小而慢的视觉推理模型的能力。通过从基础模型输出分布中减去RFT推理模型的输出分布,调整解码动态,成功激发出自我验证与自纠正等复杂行为。ProxyThinker在空间、数学及跨学科推理等挑战性视觉基准上持续提升性能,使未调优的基础模型媲美全规模RFT模型。此外,我们实现的并行化多模型协同机制,使推理速度比以往方法快达38×,为ProxyThinker的实际应用铺平道路。代码已开源:https://github.com/MrZilinXiao/ProxyThinker。
原文摘要 · Abstract (English)
Recent advancements in reinforcement learning with verifiable rewards have pushed the boundaries of the visual reasoning capabilities in large vision-language models (LVLMs). However, training LVLMs with reinforcement fine-tuning (RFT) is computationally expensive, posing a significant challenge to scaling model size. In this work, we propose ProxyThinker, an inference-time technique that enables large models to inherit the visual reasoning capabilities from small, slow-thinking visual reasoners without any training. By subtracting the output distributions of base models from those of RFT reasoners, ProxyThinker modifies the decoding dynamics and successfully elicits the slow-thinking reasoning demonstrated by the emerged sophisticated behaviors such as self-verification and self-correction. ProxyThinker consistently boosts performance on challenging visual benchmarks on spatial, mathematical, and multi-disciplinary reasoning, enabling untuned base models to compete with the performance of their full-scale RFT counterparts. Furthermore, our implementation efficiently coordinates multiple language models with parallelism techniques and achieves up to 38 $\times$ faster inference compared to previous decoding-time methods, paving the way for the practical deployment of ProxyThinker. Code is available at https://github.com/MrZilinXiao/ProxyThinker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。