TensLoRA用张量结构统一优化LoRA，提升模型适配效率。
TensLoRA: Tensor Alternatives for Low-Rank Adaptation
- 将LoRA的低秩矩阵整合为高阶张量，实现更灵活的参数共享
- 在视觉与语言任务中，相同参数量下性能优于标准LoRA
- 支持按模态调节压缩率，适合资源受限场景
低秩适配（LoRA）通过向注意力投影添加可训练的低秩矩阵，高效地微调Transformer模型。尽管有效，这些矩阵通常独立于每个注意力投影（查询、键、值）和每一层。近期扩展尝试了基于张量的联合适配，但形式有限且缺乏系统框架。我们提出TensLoRA，一个统一框架，将LoRA更新聚合为高阶张量，建模广泛的张量化低秩适配方法。该公式推广了现有张量方法，并支持模态特异性压缩率，使参数预算可根据模态和任务灵活调整。在视觉与语言基准上的实验表明，张量构造直接影响性能，有时在相似参数量下优于标准LoRA。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are considered independent for each attention projection (Query, Key, and Value) and each layer. Recent extensions have considered joint, tensor-based adaptations, but only in limited forms and without a systematic framework. We introduce TensLoRA, a unified framework that aggregates LoRA updates into higher-order tensors and models a broad family of tensor-based low-rank adaptations. Our formulation generalizes existing tensor-based methods and enables mode-specific compression rates, allowing parameter budgets to be tailored according to the modality and task. Experiments on vision and language benchmarks reveal that the tensor construction directly impacts performance, sometimes better than standard LoRA under similar parameter counts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。