混合训练让模型更准更快，省时省算力。
Mixtraining: A Better Trade-Off Between Compute and Performance
- 把自监督和有监督训练交替进行，统一在一个流程里。
- 在TinyImageNet上准确率提升8.81%，训练快1.29倍。
- 适合算力有限但想提性能的场景，支持单任务和多任务。
在标准有监督学习前加入自监督学习（SSL）已成为提升模型性能的常用策略，尤其在数据有限的情况下。然而，该方法在计算与性能间带来权衡：尽管SSL有助于表征学习，但需额外耗时的训练阶段，增加计算开销，限制了资源受限环境下的效率。为此，我们提出MixTraining，一种新框架，将多个SSL与SL训练周期在统一的混合训练阶段中交错进行，实现两种学习目标间的平滑过渡。MixTraining增强SSL与SL的协同作用以提升精度，并合并共享计算步骤以减少计算开销。该方法具备通用性，适用于单任务和多任务学习。大量实验表明，相比传统流程，MixTraining提供了更优的计算-性能权衡，在ViT-Tiny模型下于TinyImageNet上实现8.81%的绝对准确率提升（相对提升18.89%），同时训练速度最快提升1.29倍。
原文摘要 · Abstract (English)
Incorporating self-supervised learning (SSL) before standard supervised learning (SL) has become a widely used strategy to enhance model performance, particularly in data-limited scenarios. However, this approach introduces a trade-off between computation and performance: while SSL helps with representation learning, it requires a separate, often time-consuming training phase, increasing computational overhead and limiting efficiency in resource-constrained settings. To address these challenges, we propose MixTraining, a novel framework that interleaves several SSL and SL epochs within a unified mixtraining training phase, featuring a smooth transition between two learning objectives. MixTraining enhances synergy between SSL and SL for improved accuracy and consolidates shared computation steps to reduce computation overhead. MixTraining is versatile and applicable to both single-task and multi-task learning scenarios. Extensive experiments demonstrate that MixTraining offers a superior compute-performance trade-off compared to conventional pipelines, achieving an 8.81% absolute accuracy gain (18.89% relative accuracy gain) on the TinyImageNet dataset while accelerating training by up to 1.29x with the ViT-Tiny model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。