LaX让低秩模型性能逼近全参数模型，仅需一半参数。
LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
- 通过跨低秩子空间的信息流动增强模型容量。
- 在60M到1B参数模型上，性能媲美甚至超过全参数基线，参数减少2-3倍。
- 适配LoRA微调大模型，推理任务表现提升，开销极小。
训练视觉变换器（ViT）和大语言模型（LLM）等基础模型需要巨大的计算成本。低秩矩阵或张量分解提供了一种参数高效替代方案，但常因参数空间受限导致性能下降。本文提出一种简单而有效的即插即用模块——潜空间交叉（Latent Crossing, LaX），通过促进低秩子空间间的信息流动，增强低秩模型的表达能力。我们在包含ViT-Base/Large和类似LLaMA的模型（参数规模60M至1B）的预训练任务中广泛验证了LaX的有效性。结果表明，使用2-3倍更少参数的低秩模型，在性能上可达到甚至超越全参数基准。当结合低秩适配器（LoRA）用于微调LLaMA-7/13B时，LaX在算术与常识推理任务中持续提升表现，且额外开销可忽略不计。
原文摘要 · Abstract (English)
Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often downgrades performance due to the restricted parameter space. In this work, we introduce {\textbf{Latent Crossing (LaX)}} -- a simple yet effective plug-and-play module that enhances the capacity of low-rank models by enabling information flow across low-rank subspaces. We extensively validate the benefits of LaX on pre-training tasks with ViT-Base/Large and LLaMA-like models ranging from 60M to 1B parameters. LaX boosts low-rank model performance to match or exceed the full-rank baselines while using 2-3\(\times\) fewer parameters. When equipped with low-rank adapters (i.e., LoRA) for fine-tuning LLaMA-7/13B, LaX consistently improves performance on arithmetic and common sense reasoning tasks with negligible cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。