arXiv:2602.22538cs.LGcs.CL2026-02被引 5

让大模型既听懂指令又保持推理结构,不改原有思维格式。

RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format

  • 通过投影任务向量到思维标记的零空间,保留原始推理流程。
  • 用小样本校准指令注意力,提升指令遵循度且不损失推理能力。
  • 无需反向传播,适配多种模型架构与规模,特别适合智能体应用。

大型推理模型(LRMs)擅长长链推理,但常无法忠实遵循输出格式、约束或具体要求等指令。我们探究将指令微调模型(ITM)融入LRM是否能弥合这一差距。分析参数空间中的任务向量发现,关键模块的主子空间近乎正交,表明轻量级融合干扰极小。然而,我们亦证明简单融合脆弱,因LRM(含显式思维与回答段)与ITM(仅答案)存在输出格式不匹配。为此提出RAIN-Merging(基于推理感知的指令注意力引导零空间投影融合),一种无梯度方法,在保留推理格式与性能的同时增强指令遵循。首先,利用小规模推理校准集,将ITM任务向量投影至思维特殊标记前向特征的零空间,以保护LRM的结构化推理机制;其次,使用小规模指令校准集估计指令注意力,推导模块特定缩放因子,放大指令相关成分并抑制泄漏。在四个指令遵循基准和九个推理与通用能力基准上,RAIN-Merging显著提升指令遵从性,同时维持推理质量。增益在不同模型规模与架构间一致,适用于智能体场景。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) excel at a long chain of reasoning but often fail to faithfully follow instructions regarding output format, constraints, or specific requirements. We investigate whether this gap can be closed by integrating an instruction-tuned model (ITM) into an LRM. Analyzing their differences in parameter space, namely task vectors, we find that their principal subspaces are nearly orthogonal across key modules, suggesting a lightweight merging with minimal interference. However, we also demonstrate that naive merges are fragile because they overlook the output format mismatch between LRMs (with explicit thinking and response segments) and ITMs (answers-only). We introduce RAIN-Merging (Reasoning-Aware Instruction-attention guided Null-space projection Merging), a gradient-free method that integrates instruction following while preserving thinking format and reasoning performance. First, with a small reasoning calibration set, we project the ITM task vector onto the null space of forward features at thinking special tokens, which preserves the LRM's structured reasoning mechanisms. Second, using a small instruction calibration set, we estimate instruction attention to derive module-specific scaling that amplifies instruction-relevant components and suppresses leakage. Across four instruction-following benchmarks and nine reasoning & general capability benchmarks, RAIN-Merging substantially improves instruction adherence while maintaining reasoning quality. The gains are consistent across model scales and architectures, translating to improved performance in agent settings.

指令遵循推理模型模型融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。