让大模型同时保留旧知识并添加新专长，避免遗忘。
Superposition in Transformers: A Novel Way of Building Mixture of Experts
- 用自编码器在共享参数空间中叠加基础模型与微调模型的隐藏状态
- 通过B样条混合系数实现平滑切换，有效缓解灾难性遗忘
- 适合需要持续学习、多任务共存的部署场景
大语言模型在适应新任务或领域时，灾难性遗忘仍是主要挑战。传统微调常覆盖原有知识，导致原任务性能下降。我们提出Transformer中的超位置机制，利用自编码器将基础模型与微调模型的隐藏表示在共享参数空间中叠加。通过基于B样条的混合系数及根据输入数据分布自适应重构隐藏状态的自编码器，该方法有效缓解灾难性遗忘，实现‘模型内’超位置的新范式。该方法在保持原模型能力的同时，可紧凑地加入特定领域的专业知识，并支持推理时动态切换模型状态。
原文摘要 · Abstract (English)
Catastrophic forgetting remains a major challenge when adapting large language models (LLMs) to new tasks or domains. Conventional fine-tuning often overwrites existing knowledge, causing performance degradation on original tasks. We introduce Superposition in Transformers, a novel architecture that leverages autoencoders to superimpose the hidden representations of a base model and a fine-tuned model within a shared parameter space. By using B-spline-based blending coefficients and autoencoders that adaptively reconstruct hidden states based on the input data distribution, our method effectively mitigates catastrophic forgetting and enables a new paradigm of "in-model" superposition. This approach preserves original model capabilities while allowing compact domain-specific expertise to be added, and it supports dynamic switching between model states during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。