提出动态通道选择方法，在低内存下实现高效微调。
Study of Training Dynamics for Memory-Constrained Fine-Tuning
- 按架构预判层重要性，动态随机重采样通道
- 激活稀疏度达99%，权重梯度计算量减少97%
- 适合资源受限场景下的模型微调
随着模型规模增大而部署环境资源受限，高效内存训练变得愈发重要。我们提出TraDy，一种新型迁移学习方案，基于两个关键洞察：层更新的重要性与网络架构相关且可预先确定；动态随机通道选择相比静态方法能提供更优的梯度近似。我们引入一种动态通道选择机制，在预选层内跨周期随机重采样通道。大量实验表明，TraDy在多种下游任务和架构上达到当前最优性能，同时严格满足内存约束，实现高达99%的激活稀疏度、95%的权重导数稀疏度，以及权重导数计算中97%的浮点运算减少。
原文摘要 · Abstract (English)
Memory-efficient training of deep neural networks has become increasingly important as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependent and determinable a priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state-of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction in FLOPs for weight derivative computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。