arXiv:2601.08209cs.CL2026-01

用生成式方法高效注入私有领域知识,不改主模型还能保持通用能力。

Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Models

  • 将私域知识作为辅助模态,通过轻量级隐空间接口注入冻结模型。
  • 在两个科学问答基准上优于检索和微调基线,同时保留通用能力。
  • 适合需持续更新私有知识的高风险领域如生物医药、材料科学。

在材料科学、生物医学和金融等高风险领域,大语言模型部署需注入专有、快速演进且公共预训练中未充分覆盖的私有知识。现有主流方法各有缺陷:微调成本高,频繁更新易引发灾难性遗忘与通用能力退化;检索增强生成(RAG)虽保持基础模型不变,但在私有专业语料中仍因分块导致证据碎片化、检索不匹配及长上下文压力而表现脆弱。受多模态模型对齐异构模态思想启发,本文提出生成增强生成(GAG),将私有专长视为辅助模态,通过紧凑恒定预算的隐空间接口注入冻结的基础模型。具体而言,GAG从轻量级领域专家中提炼条件化问题的知识,生成多槽隐记忆,通过每槽跨层融合整合多层专家信号,并利用门控残差投影将其对齐至冻结基模型,支持可扩展的多领域部署与可靠的选择性激活。在涵盖催化材料与免疫佐剂两个私有领域QA基准及通用领域查询的统一评估中,GAG在专业问答任务上持续优于强检索与参数高效微调基线,同时保持通用能力,实现高可靠性路由,并具备良好的效率-效果权衡。代码与数据集见补充材料,代码公开于 https://github.com/360CVGroup/GAG。

原文摘要 · Abstract (English)

In domains such as materials science, biomedicine, and finance, high-stakes deployment of large language models (LLMs) requires injecting private, domain-specific knowledge that is proprietary, fast-evolving, and under-represented in public pretraining. However, the two dominant paradigms for private knowledge injection each have clear drawbacks: fine-tuning is expensive to iterate under continual updates that can induce catastrophic forgetting and general-capability regression; retrieval-augmented generation (RAG) keeps the base model intact but remains brittle in specialized private corpora due to chunk-induced evidence fragmentation, retrieval mismatch, and long-context pressure. Inspired by how multimodal LLMs align heterogeneous modalities into a shared semantic space, we propose Generation-Augmented Generation (GAG), which treats private expertise as an auxiliary modality and injects it into a frozen base model through a compact, constant-budget latent interface. Concretely, GAG distills question-conditioned specialist knowledge from lightweight domain experts into multi-slot latent memories, integrates multi-layer expert signals via per-slot cross-layer fusion, and aligns them to the frozen base model through gated residual projection, while supporting scalable mixed-domain deployment with reliable selective activation. In a unified mixed-domain evaluation spanning two scientific private-domain QA benchmarks (catalytic materials and immunology adjuvant) together with general-domain queries, GAG consistently outperforms strong retrieval-based and parameter-efficient fine-tuning baselines on specialist QA, while preserving general-domain capability, achieving highly reliable routing, and offering a favorable efficiency--effectiveness trade-off. Code and datasets are provided in the supplementary material. Code is publicly available at https://github.com/360CVGroup/GAG.

知识注入LLM安全私有数据RAG改进

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。