arXiv:2504.01792cs.CV2025-04被引 7

让视觉Transformer原生处理不同分辨率图像,提升细节与上下文感知能力。

UniViTAR: Unified Vision Transformer with Native Resolution

  • 采用渐进式训练策略,从固定分辨率过渡到原生分辨率微调。
  • 在0.3B到1B参数规模下,跨多数据集验证了模型有效性。
  • 适合需要高保真视觉建模的多模态应用,如视频理解与生成。

传统视觉Transformer为简化视觉建模,将输入统一为固定分辨率,常忽略自然视觉数据的多样性,损害空间上下文保真度。尽管已有初步探索尝试原生分辨率建模,但缺乏从视觉表征角度的系统分析。为此,我们提出UniViTAR,一个专为统一视觉模态与原生分辨率场景设计的同质化视觉基础模型家族。框架首先通过集成多种先进组件对基础架构进行升级;在此基础上,引入渐进式训练范式,策略性结合两项核心机制:(1) 分辨率课程学习,从固定分辨率预训练逐步过渡至原生分辨率微调,利用ViT对可变长度序列的天然适应性;(2) 通过批间图像-视频切换实现视觉模态自适应,平衡计算效率与增强的时间推理能力。同时,混合训练框架进一步融合基于sigmoid的对比损失与来自冻结教师模型的特征蒸馏,加速早期收敛。最终,仅在公开数据集上训练,跨多个模型尺度(0.3B至1B)的广泛实验验证了其有效性。

原文摘要 · Abstract (English)

Conventional Vision Transformer simplifies visual modeling by standardizing input resolutions, often disregarding the variability of natural visual data and compromising spatial-contextual fidelity. While preliminary explorations have superficially investigated native resolution modeling, existing approaches still lack systematic analysis from a visual representation perspective. To bridge this gap, we introduce UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario in the era of multimodal. Our framework first conducts architectural upgrades to the vanilla paradigm by integrating multiple advanced components. Building upon these improvements, a progressive training paradigm is introduced, which strategically combines two core mechanisms: (1) resolution curriculum learning, transitioning from fixed-resolution pretraining to native resolution tuning, thereby leveraging ViT's inherent adaptability to variable-length sequences, and (2) visual modality adaptation via inter-batch image-video switching, which balances computational efficiency with enhanced temporal reasoning. In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model, thereby accelerating early-stage convergence. Finally, trained exclusively on public datasets, externsive experiments across multiple model scales from 0.3B to 1B demonstrate its effectiveness.

视觉Transformer原生分辨率多模态建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。