用张量增强技术让小设备也能高效用大模型特征。
Learning from Offline Foundation Features with Tensor Augmentations
- 对缓存的图像特征做张量增强，避免存储大量增广数据。
- 训练速度提升37倍，显存减少26倍，适合资源受限场景。
- 可处理高分辨率图像，且效果优于直接微调大模型。
我们提出一种名为LOFF-TA的高效训练方法，用于在计算资源有限的环境下利用冻结的预训练基础模型的特征。该方法在缓存的基础模型特征上训练一个轻量级分类器，实现最高37倍的训练加速和最高26倍的显存降低。由于增广图像的特征数量庞大难以存储，但又必须通过增广来训练，我们提出对原始未增广图像的特征应用张量增强。这使得无论基础模型大小，都能在低算力条件下有效使用其能力。此外，该方法可无额外计算成本地处理高分辨率图像，在某些情况下性能优于直接微调基础模型。
原文摘要 · Abstract (English)
We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to $37\times$ faster training and up to $26\times$ reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。