用自适应基向量压缩视觉令牌，让大模型更高效
Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning
- 将图像令牌视为语义空间向量，定义最小基向量集
- 大模型仅需少量基向量即可保留图像语义，符合缩放定律
- 轻量模块实现冗余令牌聚类，适合高分辨率视觉任务
本文研究模型容量与维持图像语义所需的最少视觉令牌数之间的基本关系。受最小描述长度原理启发，我们将图像令牌重新诠释为视觉语义空间中的向量，并将图像的内在语义复杂度定义为能够张成该空间的最小基向量集合。基于此观点，我们提出正交过滤（Orthogonal Filtering）模块，可自适应地将冗余令牌聚类为一组紧凑的正交基。在多种ViT模型上进行的广泛实验表明，存在一致的令牌-模型缩放律：模型越大，所需覆盖视觉语义空间的令牌越少。此外，我们还构建了一个视觉长上下文数据集。
原文摘要 · Abstract (English)
This paper investigates the fundamental relationship between model capacity and the minimal number of visual tokens required to preserve image semantics. Inspired by the Minimum Description Length principle, we reinterpret image tokens as vectors in a visual semantic space and define the intrinsic semantic complexity of an image as the smallest set of basis vectors needed to span this space. Building on this perspective, we propose Orthogonal Filtering, a lightweight module that adaptively clusters redundant tokens into a compact set of orthogonal bases. Through extensive experiments across a range of ViT models, we reveal a consistent token, model scaling law: larger models require significantly fewer tokens to span visual semantic space. Besides, we also contribute a visual long-context dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。