用低秩适配动态替换训练中后期的视觉Transformer参数，大幅降本增效。
PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters
- 训练中后期切换为低秩适配，按层自适应设置秩。
- 参数量降至10%，每轮训练时间减少50%，吞吐提升3倍。
- 适合资源受限场景下大模型高效微调，尤其关注显存优化者。
训练包含数百万至数十亿参数的大模型耗时耗力，需要大量计算与内存。研究发现，模型权重在训练初期变化最大，后期趋于稳定，表明后续更新可用低秩矩阵近似。为此，我们提出在ViT-Large模型上识别部分收敛状态，并动态从全参数训练切换至低秩适配（LoRA）。该方法通过用户定义的超参数确定切换点，并根据各模块层的收敛程度分配特定秩。实验显示，该方法在保持模型精度的同时，将可训练参数减少至原规模的10%，实现吞吐量提升3倍，每轮训练平均时间缩短50%，且显存消耗降低20%。
原文摘要 · Abstract (English)
Training large models ranging from millions to billions of parameters is highly resource-intensive, requiring significant time, compute, and memory. It is observed that most of the learning (higher change in weights) takes place in the earlier stage of the training loop. As training progresses, these changes stabilize, suggesting that the resulting updates may be amenable to approximation using low intrinsic-rank matrices. Therefore, we propose an approach to identify such states of partial convergence and dynamically switch from full parameter training to Low Rank Adaptation (LoRA) on the ViT-Large model. We introduce a flexible approach that leverages user-defined hyperparameters to determine the switching point and assign a rank specific to each module layer based on its level of convergence. Experimental results show that this approach preserves model accuracy while reducing the number of trainable parameters to 10% of its original size, resulting in a 3x improvement in throughput, and a 1.5x reduction in average training time per epoch while also reducing GPU memory consumption by 20%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。