通过渐进式训练实现视觉自回归模型的高效部署,支持运行时动态调整深度。
Progressive Supernet Training for Efficient Visual Autoregressive Modeling
- 从30层网络中采样多尺度子网,早期用全深网络,后期用浅层子网共享权重。
- 在ImageNet上,16层子网仅损失微小质量(FID 2.05)却省40%-65%内存。
- 单模型支持零成本深度切换,适合从高清到极低功耗的各种应用场景。
视觉自回归(VAR)模型通过“下一尺度”预测减少推理步数,但渐进式多尺度生成导致累积键值缓存带来巨大内存开销,限制实际部署。我们观察到VAR存在尺度-深度非对称依赖:早期尺度对网络深度极为敏感,后期则对深度削减具有鲁棒性。受此启发,我们提出VARiant:通过等距采样,从原始30层的VAR-d30网络中选取16至2层的多个子网。早期尺度使用完整网络,后期尺度采用子网。子网与全网络共享权重,实现单模型内灵活深度调节。然而,权重共享可能引发优化冲突。为此,我们提出一种渐进式训练策略,突破固定比例训练下的生成质量帕累托前沿,在子网与全网络间实现联合最优。ImageNet实验表明,相比预训练的VAR-d30(FID 1.95),VARiant-d16和VARiant-d8分别达到近似质量(FID 2.05/2.12),内存降低40%-65%。VARiant-d2实现3.5倍加速和80%内存缩减,质量略有下降(FID 2.97)。在部署方面,VARiant的单模型架构支持零成本运行时深度切换,提供从高保真到极致效率的灵活选择,适配多样化应用场景。
原文摘要 · Abstract (English)
Visual Auto-Regressive (VAR) models significantly reduce inference steps through the "next-scale" prediction paradigm. However, progressive multi-scale generation incurs substantial memory overhead due to cumulative KV caching, limiting practical deployment. We observe a scale-depth asymmetric dependency in VAR: early scales exhibit extreme sensitivity to network depth, while later scales remain robust to depth reduction. Inspired by this, we propose VARiant: by equidistant sampling, we select multiple subnets ranging from 16 to 2 layers from the original 30-layer VAR-d30 network. Early scales are processed by the full network, while later scales utilize subnet. Subnet and the full network share weights, enabling flexible depth adjustment within a single model. However, weight sharing between subnet and the entire network can lead to optimization conflicts. To address this, we propose a progressive training strategy that breaks through the Pareto frontier of generation quality for both subnets and the full network under fixed-ratio training, achieving joint optimality. Experiments on ImageNet demonstrate that, compared to the pretrained VAR-d30 (FID 1.95), VARiant-d16 and VARiant-d8 achieve nearly equivalent quality (FID 2.05/2.12) while reducing memory consumption by 40-65%. VARiant-d2 achieves 3.5 times speedup and 80% memory reduction at moderate quality cost (FID 2.97). In terms of deployment, VARiant's single-model architecture supports zero-cost runtime depth switching and provides flexible deployment options from high quality to extreme efficiency, catering to diverse application scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。