arXiv:2601.11258cs.LGcs.AI2026-01ACL被引 3

用强化学习技能提升大模型知识应用能力,实现高效持续适应。

Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation

  • 从强化学习中提取通用技能向量,线性注入模型提升知识使用能力。
  • 在SQuAD上比顶尖自编辑微调方法高9.9分,在ToolBench上零样本成功率提升10.3点。
  • 技能向量可跨领域迁移,适用于长文本问答与多工具任务,适合需持续更新的场景。

大型语言模型面临“知识截止”问题,其固定参数无法直接内化新信息。尽管监督微调(SFT)常用于更新知识,但往往只改变事实内容,未能有效提升模型对新知识的问答或决策使用能力。强化学习虽能获取推理技能,但计算成本高,难以实现高效在线适应。我们发现SFT与RL引起的参数更新近乎正交。基于此,提出参数化技能迁移(PaST)框架,支持模块化技能迁移。通过从源域提取无领域依赖的技能向量,可在轻量级SFT后线性注入目标模型,以增强知识操作能力。在知识融入型问答(SQuAD、LooGLE)和代理工具使用基准(ToolBench)上的实验表明,该方法效果显著:在SQuAD上较最优自编辑SFT基线最高提升9.9分;在长上下文问答任务LooGLE上绝对准确率提升8.0分;在ToolBench上平均零样本成功率提升10.3点,且各类工具均表现稳定增益,体现强可扩展性与跨领域迁移能力。

原文摘要 · Abstract (English)

Large Language Models (LLMs) face the "knowledge cutoff" challenge, where their frozen parametric memory prevents direct internalization of new information. While Supervised Fine-Tuning (SFT) is commonly used to update model knowledge, it often updates factual content without reliably improving the model's ability to use the newly incorporated information for question answering or decision-making. Reinforcement Learning (RL) is essential for acquiring reasoning skills; however, its high computational cost makes it impractical for efficient online adaptation. We empirically observe that the parameter updates induced by SFT and RL are nearly orthogonal. Based on this observation, we propose Parametric Skill Transfer (PaST), a framework that supports modular skill transfer for efficient and effective knowledge adaptation. By extracting a domain-agnostic Skill Vector from a source domain, we can linearly inject knowledge manipulation skills into a target model after it has undergone lightweight SFT on new data. Experiments on knowledge-incorporation QA (SQuAD, LooGLE) and agentic tool-use benchmarks (ToolBench) demonstrate the effectiveness of our method. On SQuAD, PaST outperforms the state-of-the-art self-editing SFT baseline by up to 9.9 points. PaST further scales to long-context QA on LooGLE with an 8.0-point absolute accuracy gain, and improves zero-shot ToolBench success rates by +10.3 points on average with consistent gains across tool categories, indicating strong scalability and cross-domain transferability of the Skill Vector.

大模型强化学习知识更新技能迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。