用共享的影子模块实现更高效的模型微调,性能超越传统方法。
ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

- 引入深度共享的影子模块,在每层进行集中式状态优化。
- 在生成与理解任务上,参数量相近时效果优于或等同于LoRA和DoRA。
- 支持跨层复用和边缘部署,适合资源受限场景。
参数高效微调(PEFT)通过仅训练少量特定任务参数而冻结预训练主干,降低大语言模型微调成本。现有方法如低秩适应(LoRA)通过直接对各权重插入独立低秩扰动实现适配,造成适应过程分散。我们提出ShadowPEFT,一种集中式PEFT框架,通过深度共享的影子模块在每一Transformer层执行层级精炼。每个层维护并重复演化一个平行影子状态,逐步丰富隐藏状态。该设计将适配从分布式权重空间扰动转为共享层空间精炼。由于影子模块与主干解耦,可跨深度复用、独立预训练,并可选择性以脱离模式部署,适用于边缘计算场景。在生成与理解基准测试中,ShadowPEFT在相近可训练参数预算下表现匹配或优于LoRA和DoRA。对影子预训练、跨数据集迁移、参数缩放、推理延迟及系统级评估的额外分析表明,集中式层空间适配是传统低秩PEFT的有力且灵活替代方案。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。