arXiv:2507.19360cs.CV2025-07ICCV被引 8

一个训练流程生成多种尺寸视觉变压器,适配不同设备资源。

EA-ViT: Efficient Adaptation for Elastic Vision Transformer

  • 用嵌套弹性结构动态调整模型大小,支持多尺度部署。
  • 单次训练可生成多个子模型,性能优于独立训练的基线模型。
  • 适合边缘设备、移动终端等算力受限场景使用。

视觉变换器(ViTs)在计算机视觉中已成为基础模型,具备良好的泛化能力和下游任务适应性。然而,在多样化的资源约束下部署ViTs通常需要为不同规模重新训练多个模型,耗时且耗能。为此,我们提出一种高效的ViT自适应框架,使单一适应过程即可生成多种尺寸的模型,以满足不同平台的资源需求。该方法分为两个阶段:第一阶段,通过嵌套弹性架构增强预训练的ViT,实现对MLP扩展比例、注意力头数、嵌入维度和网络深度的结构灵活性;为保留预训练知识并确保稳定适应,采用渐进式课程训练策略逐步增加弹性。第二阶段,设计轻量级路由器,根据计算预算和下游任务需求选择子模型。路由器初始化于通过定制化NSGA-II算法获得的帕累托最优配置,并与主干网络联合优化。在多个基准上的大量实验表明,EA-ViT具有优异的有效性和通用性。代码已公开于https://github.com/zcxcf/EA-ViT。

原文摘要 · Abstract (English)

Vision Transformers (ViTs) have emerged as a foundational model in computer vision, excelling in generalization and adaptation to downstream tasks. However, deploying ViTs to support diverse resource constraints typically requires retraining multiple, size-specific ViTs, which is both time-consuming and energy-intensive. To address this issue, we propose an efficient ViT adaptation framework that enables a single adaptation process to generate multiple models of varying sizes for deployment on platforms with various resource constraints. Our approach comprises two stages. In the first stage, we enhance a pre-trained ViT with a nested elastic architecture that enables structural flexibility across MLP expansion ratio, number of attention heads, embedding dimension, and network depth. To preserve pre-trained knowledge and ensure stable adaptation, we adopt a curriculum-based training strategy that progressively increases elasticity. In the second stage, we design a lightweight router to select submodels according to computational budgets and downstream task demands. Initialized with Pareto-optimal configurations derived via a customized NSGA-II algorithm, the router is then jointly optimized with the backbone. Extensive experiments on multiple benchmarks demonstrate the effectiveness and versatility of EA-ViT. The code is available at https://github.com/zcxcf/EA-ViT.

视觉变压器弹性模型自适应部署轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。