一个可插拔的推理模块,让冻结的大模型轻松获得专业推理能力。
Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs
- 设计可组合的推理模块,通过解耦训练生成逐标记指导信号。
- 在数学推理和机器翻译任务上超越现有微调方法,效果媲美大模型。
- 支持跨模型大小与领域迁移,适合希望低成本增强推理的开发者。
大型语言模型(LLMs)展现出卓越的通用能力,但提升如推理等技能常需大量计算资源,且可能损害泛化性。参数高效微调(PEFT)虽更省资源,却因架构依赖需为每种模型重新训练。为此,我们提出通用推理器(UniR)——一个模块化、可组合、即插即用的推理模块,可与更大的冻结LLM配合使用,在共享或对齐的标记空间中提供专用推理能力。具体而言,UniR将轨迹级奖励分解为独立训练的推理模块,利用可验证奖励进行解耦训练,有效将轨迹级信号转化为标记级指导。训练完成后,仅在推理时通过将模块输出逻辑值叠加至主干模型即可融合。这种加法结构支持模块组合:多个针对不同任务训练的UniR模块可通过逻辑值相加联合应用,实现复杂推理。此外,UniR表现出弱到强的泛化能力,即在小模型上训练的推理模块可有效引导同家族更大模型,且跨领域适用于视觉语言模型与医学推理。在数学推理与机器翻译实验中,UniR性能超越现有微调方法。代码已开源:https://github.com/hangeol/UniR。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable general capabilities, but enhancing skills such as reasoning often demands substantial computational resources and may compromise generalization. While Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious alternative, they typically require retraining for each LLM backbone due to architectural dependencies. To address these challenges, we propose Universal Reasoner (UniR)-a modular, composable, and plug-and-play reasoning module that can be used with larger frozen LLMs to provide specialized reasoning capabilities with a shared or aligned token space. Specifically, UniR decomposes the reward into a standalone reasoning module trained in a decoupled manner using verifiable rewards, effectively translating trajectory-level signals into token-level guidance. Once trained, UniR is combined with frozen LLMs at inference time by simply adding its output logits to those of the backbone. This additive structure enables modular composition: multiple UniR modules trained for different tasks can be jointly applied by summing their logits, enabling complex reasoning via composition. Furthermore, UniR demonstrates weak-to-strong generalization, where reasoning modules trained on smaller models effectively guide much larger LLMs in the same model family, and generalize across domains such as in vision language models and medical reasoning. Experiments on mathematical reasoning and machine translation show that UniR surpasses existing fine-tuning methods. Code is open-sourced at https://github.com/hangeol/UniR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。