arXiv:2511.11683cs.LGcs.AI2025-11AAAI被引 3

将ViT转为分层知识密度超网络,灵活适配不同资源需求。

Stratified Knowledge-Density Super-Network for Scalable Vision Transformers

  • 通过加权PCA压缩注意力权重,集中关键知识于少数参数。
  • 在ImageNet上,小模型比传统剪枝方法提升3.2%准确率。
  • 适合需要多尺寸部署的视觉模型开发者使用。

为应对在不同资源约束下训练和部署多个视觉变换器(ViT)模型的高成本与低效问题,本文提出将预训练的ViT转化为分层知识密度超网络,使知识在权重中呈层次化组织,从而可灵活提取保留最大知识的子网络以适应不同模型规模。提出加权PCA注意力收缩(WPAC)方法,对中间特征进行逐标记加权主成分分析,并将得到的变换及逆变换矩阵注入相邻层,在保持原网络功能的同时增强知识紧凑性。为进一步促进分层知识组织,提出渐进重要性感知丢弃(PIAD),逐步评估权重组重要性,更新重要性感知丢弃列表,并在该丢弃策略下训练超网络以强化知识分层。实验表明,WPAC在知识集中度上优于现有剪枝标准,结合PIAD后,其性能可媲美最先进的模型压缩与扩展方法。

原文摘要 · Abstract (English)

Training and deploying multiple vision transformer (ViT) models for different resource constraints is costly and inefficient. To address this, we propose transforming a pre-trained ViT into a stratified knowledge-density super-network, where knowledge is hierarchically organized across weights. This enables flexible extraction of sub-networks that retain maximal knowledge for varying model sizes. We introduce \textbf{W}eighted \textbf{P}CA for \textbf{A}ttention \textbf{C}ontraction (WPAC), which concentrates knowledge into a compact set of critical weights. WPAC applies token-wise weighted principal component analysis to intermediate features and injects the resulting transformation and inverse matrices into adjacent layers, preserving the original network function while enhancing knowledge compactness. To further promote stratified knowledge organization, we propose \textbf{P}rogressive \textbf{I}mportance-\textbf{A}ware \textbf{D}ropout (PIAD). PIAD progressively evaluates the importance of weight groups, updates an importance-aware dropout list, and trains the super-network under this dropout regime to promote knowledge stratification. Experiments demonstrate that WPAC outperforms existing pruning criteria in knowledge concentration, and the combination with PIAD offers a strong alternative to state-of-the-art model compression and model expansion methods.

视觉Transformer模型压缩知识密度超网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。