一种统一框架,让模型压缩与高效微调共存,适合资源受限场景。
Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression
- 将权重分解为共享基矩阵和混合投影,实现多任务统一
- 混合参数少于200,压缩后仍保持性能领先4-5%
- 兼顾模型压缩与高效微调,适合边缘设备部署
参数重组(PR)方法旨在高效组合神经网络权重,用于参数高效微调(PEFT)和模型压缩(MC)等应用。现有方法多聚焦单一任务,难以协同部署。例如大型模型需同时压缩并快速适应新场景,但传统PEFT仍含数百万参数,在边缘设备上占比过高。为此,本文提出系数门控的权重重组方法(CRISP),通过将预训练权重分解为基矩阵与混合投影,实现多任务统一。共享基矩阵支持模型压缩,而小规模混合权重(实验中少于200)可实现高效微调。实验表明,CRISP在双任务场景下优于先前方法4-5%,在单任务PEFT上比当前最优高出1.5%,在联合任务中提升1%。代码已开源。
原文摘要 · Abstract (English)
Parameter Recombination (PR) methods aim to efficiently compose the weights of a neural network for applications like Parameter-Efficient FineTuning (PEFT) and Model Compression (MC), among others. Most methods typically focus on one application of PR, which can make composing them challenging. For example, when deploying a large model you may wish to compress the model and also quickly adapt to new settings. However, PEFT methods often can still contain millions of parameters. This may be small compared to the original model size, but can be problematic in resource constrained deployments like edge devices, where they take a larger portion of the compressed model's parameters. To address this, we present Coefficient-gated weight Recombination by Interpolated Shared basis Projections (CRISP), a general approach that seamlessly integrates multiple PR tasks within the same framework. CRISP accomplishes this by factorizing pretrained weights into basis matrices and their component mixing projections. Sharing basis matrices across layers and adjusting its size enables us to perform MC, whereas the mixer weight's small size (fewer than 200 in some experiments) enables CRISP to support PEFT. Experiments show CRISP outperforms methods from prior work capable of dual-task applications by 4-5\% while also outperforming the state-of-the-art in PEFT by 1.5\% and PEFT+MC combinations by 1\%. Our code is available on the repository: https://github.com/appledora/CRISP-CVPR26.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。