让视觉模型自动调整图像分块大小，提升精度与速度。
DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models
- 用可学习的区域得分和分位数划分，动态生成不同大小的图像块。
- 小模型用DART后性能媲美大模型，推理速度几乎翻倍。
- 适合需要高分辨率细节的多模态、机器人和视频任务。
标准大规模视觉模型（如ViT、Vim）使用的无内容感知固定网格分块方式存在根本性性能瓶颈，导致在捕捉细粒度信息与冗余计算之间难以兼顾。为此，我们提出完全可微的动态自适应区域分块器DART。DART通过可学习的区域得分和基于分位数的划分机制，生成尺寸不一的内容感知图像块，智能地在信息丰富区域分配更高分块密度。该方法效果显著：使配备DART的DeiT-Small（22M参数）性能达到原DeiT-Base（86M参数）水平，同时实现近两倍的推理速度。该自适应分块原则在密集预测和时空视频任务中也展现出普适优势。我们认为，从源头解决分块瓶颈，自适应分块是构建下一代高效多模态基础模型的关键组件，适用于机器人、内容生成等领域。代码已开源：https://github.com/HCPLab-SYSU/DART。
原文摘要 · Abstract (English)
The content-agnostic, fixed-grid tokenizers used by standard large-scale vision models like Vision Transformer (ViT) and Vision Mamba (Vim) represent a fundamental performance bottleneck, creating a trade-off between capturing fine-grained detail and suffering from redundant computation. To resolve this dilemma, we introduce DART, a fully differentiable Dynamic Adaptive Region Tokenizer. DART employs learnable region scores and quantile-based partitioning to create content-aware patches of varying sizes, intelligently allocating a higher token density to information-rich regions. The impact of this approach is profound: it unlocks a more intelligent scaling paradigm, where a DART-equipped DeiT-Small (22M parameters) matches the performance of a DeiT-Base (86M) with nearly double the inference speed by efficiently capturing high-resolution details in key regions. Furthermore, the principle of adaptive tokenization proves its generality with clear benefits in dense prediction and spatiotemporal video tasks. We argue that by resolving the tokenizer bottleneck at its source, adaptive tokenization is a key component for building the next generation of more efficient and capable foundation models for multimodal AI, robotics, and content generation. Code is available at https://github.com/HCPLab-SYSU/DART.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。