用压缩感知理论提升大模型微调效率与表达力
CoSA: Compressed Sensing-Based Adaptation of Large Language Models
- 通过随机投影矩阵和可学习核心表示权重更新
- 10项任务上性能媲美或超越现有高效微调方法
- 适合需要高表达力的多尺度模型适配场景
参数高效微调（PEFT）已成为无需更新全部参数即可适应大语言模型的实用范式。现有方法如LoRA和PiSSA依赖权重更新的低秩分解，但低秩假设可能限制表达能力，尤其在奇异值分布均匀的任务特定适配场景中。为此，我们提出基于压缩感知理论的CoSA（Compressed Sensing-Based Adaptation）。CoSA不将权重更新限制在低秩子空间，而是通过固定随机投影矩阵与紧凑可学习核心来表达。我们从理论上分析了CoSA作为合成过程的性质，证明权重更新可被紧凑编码至低维空间并通过随机投影重构。大量实验表明，CoSA为高效且具表达力的多尺度模型适配提供了原则性视角。我们在10个不同任务上评估，涵盖自然语言理解与生成，使用来自RoBERTa、Llama和Qwen系列的5种不同规模模型。在所有设置下，CoSA均表现一致优于或媲美当前最优的PEFT方法。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a practical paradigm for adapting large language models (LLMs) without updating all parameters. Most existing approaches, such as LoRA and PiSSA, rely on low-rank decompositions of weight updates. However, the low-rank assumption may restrict expressivity, particularly in task-specific adaptation scenarios where singular values are distributed relatively uniformly. To address this limitation, we propose CoSA (Compressed Sensing-Based Adaptation), a new PEFT method extended from compressed sensing theory. Instead of constraining weight updates to a low-rank subspace, CoSA expresses them through fixed random projection matrices and a compact learnable core. We provide a formal theoretical analysis of CoSA as a synthesis process, proving that weight updates can be compactly encoded into a low-dimensional space and mapped back through random projections. Extensive experimental results show that CoSA provides a principled perspective for efficient and expressive multi-scale model adaptation. Specifically, we evaluate CoSA on 10 diverse tasks, including natural language understanding and generation, employing 5 models of different scales from RoBERTa, Llama, and Qwen families. Across these settings, CoSA consistently matches or outperforms state-of-the-art PEFT methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。