arXiv:2601.13580cs.LGcs.AI2026-01

将Transformer层当作可移植的模块,实现高效隐私保护的领域适配。

Neural Organ Transplantation (NOT): Checkpoint-Based Modular Adaptation for Transformer Models

  • 从预训练模型中提取连续层作为独立模块,单独训练后可直接移植到新模型。
  • 在3种不同规模模型上,困惑度比LoRA低一个数量级,训练速度更快。
  • 适合需要快速适配且不共享原始数据的场景,如私密领域知识迁移。

我们提出神经器官移植(NOT),一种模块化适配框架,使训练好的Transformer层可作为可复用的、可移植的检查点用于领域适应。与传统微调方法将参数紧耦合于特定模型和训练数据不同,NOT从预训练模型中提取连续层块(“供体器官”),在特定领域数据上独立训练,并保存为独立检查点文件,可在无需原始训练数据的情况下移植到兼容的接收模型中。在三种仅解码器的Transformer架构(124M至20B参数,GPT-2、TinyLlama、GPT-OSS)上实验表明,供体移植显著优于现有方法,困惑度比LoRA降低一个数量级,且训练速度更快。该方法表现出位置依赖性,早期插入位置效果最佳。在十亿参数级跨域迁移中揭示了意外的正则化收益。结果表明,Transformer中间层可支持解码器架构的高效模块化迁移,通过检查点分发实现隐私保护的知识共享。目前该方法仅适用于仅解码器模型;对编码器架构的初步实验显示效果下降。

原文摘要 · Abstract (English)

We introduce Neural Organ Transplantation (NOT), a modular adaptation framework that enables trained transformer layers to function as reusable transferable checkpoints for domain adaptation. Unlike conventional fine-tuning approaches that tightly couple trained parameters to specific model instances and training data, NOT extracts contiguous layer subsets ("donor organs") from pre-trained models, trains them independently on domain-specific data, and saves them as standalone checkpoint files that can be transplanted into compatible recipient models without access to the original training data. Through experiments on three decoder-only transformer architectures spanning 124M to 20B parameters (GPT-2, TinyLlama, and GPT-OSS), we demonstrate that donor transplantation substantially outperforms existing adaptation methods, achieving an order-of-magnitude improvement in perplexity over LoRA while training significantly faster. The method exhibits position dependence, with early insertion positions yielding optimal results. Cross-domain transfer at billion-parameter scale reveals unexpected regularization benefits. These findings demonstrate that transformer middle layers can support efficient modular transfer for decoder-only architectures, enabling privacy-preserving expertise sharing through checkpoint distribution. We note that this approach is currently limited to decoder-only models; preliminary experiments on encoder-based architectures show reduced effectiveness.

模型迁移模块化隐私保护Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。