通过分层共享参数设计,用极少量参数实现大模型高效微调。
SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning

- 采用夹心结构硬共享权重,平衡中间层适配器。
- 参数仅占基线10%,在理解任务上超越proPETL。
- 适合资源受限场景下的大模型轻量化微调。
尽管参数高效微调(PEFT)已显著降低大语言模型适配的硬件成本,但现有方法或采用全层均匀共享参数导致收敛延迟,或依赖动态掩码增加计算开销。本文提出SAPE(Sandwich Adapters for Parameter Efficiency),基于夹心式硬权重共享拓扑,将中间Transformer层经由均衡共享组适配器路由,同时严格隔离输入嵌入与最终投影边界变换,防止梯度干扰。该设计大幅降低内存消耗,并消除动态共享方法的计算开销。在编码器和因果解码器架构上的大量实验表明,SAPE在低参数环境下达到领先性能:在自然语言理解任务中,于RoBERTa-large上以仅10%的参数预算超越proPETL;在LLaMA-3.2(3B)的生成与常识推理任务中,于约0.6M参数约束下,相较AdaLoRA提升+4.85%(GSM8K)和+3.11%(CommonsenseQA)。通过系统拓扑消融分析,揭示出硬共享存在固有容量权衡:虽强正则化语义泛化能力,但轻微弱化多步算术推理所需的尖锐层间转换。
原文摘要 · Abstract (English)
While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。