arXiv:2510.18431cs.CVcs.AI2025-10中稿 · IEEE Transactions …

通过增量层共享参数,低成本高效扩展ViT模型性能。

ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters

  • 在预训练ViT基础上插入新层,共享权重并加微调模块。
  • 深度翻倍的DeiT-Base模型准确率提升7.42%,训练时间减少三分之二。
  • 适合需要快速扩展模型又不想重训的研究者和工程师。

视觉变换器(ViTs)的规模越大,性能通常越好,但训练成本高昂。为解决此问题,本文提出ScaleNet,一种基于预训练模型的高效扩展方法。与从头训练不同,ScaleNet通过向预训练ViT插入新层实现模型扩展,采用逐层权重共享以保持参数效率。每新增层与其对应预训练层共享参数张量,并引入少量可调适配参数,通过并行适配模块使每个共享张量保持独立优化。ImageNet-1K实验表明,2×深度扩展的DeiT-Base模型,相比从头训练,准确率提升7.42%,训练周期仅需三分之一。该方法在目标检测等下游任务中也表现出显著潜力。

原文摘要 · Abstract (English)

Recent advancements in vision transformers (ViTs) have demonstrated that larger models often achieve superior performance. However, training these models remains computationally intensive and costly. To address this challenge, we introduce ScaleNet, an efficient approach for scaling ViT models. Unlike conventional training from scratch, ScaleNet facilitates rapid model expansion with negligible increases in parameters, building on existing pretrained models. This offers a cost-effective solution for scaling up ViTs. Specifically, ScaleNet achieves model expansion by inserting additional layers into pretrained ViTs, utilizing layer-wise weight sharing to maintain parameters efficiency. Each added layer shares its parameter tensor with a corresponding layer from the pretrained model. To mitigate potential performance degradation due to shared weights, ScaleNet introduces a small set of adjustment parameters for each layer. These adjustment parameters are implemented through parallel adapter modules, ensuring that each instance of the shared parameter tensor remains distinct and optimized for its specific function. Experiments on the ImageNet-1K dataset demonstrate that ScaleNet enables efficient expansion of ViT models. With a 2$\times$ depth-scaled DeiT-Base model, ScaleNet achieves a 7.42% accuracy improvement over training from scratch while requiring only one-third of the training epochs, highlighting its efficiency in scaling ViTs. Beyond image classification, our method shows significant potential for application in downstream vision areas, as evidenced by the validation in object detection task.

ViT扩展参数共享高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。