用空间与语义分离法，让模型同时实现高质量重建和高精度分类。
Learning Sparse Visual Representations via Spatial-Semantic Factorization
- 将视觉特征分解为语义概念与空间分布的低秩乘积
- 仅用16个稀疏令牌即达2.60 FID重建质量与79.10%图像分类准确率
- 适合需要兼顾生成与判别能力的视觉任务研究者
自监督学习在语义理解与图像重建之间存在根本矛盾：高层语义方法（如DINO）依赖全局令牌，为对齐增强而强制位置不变性，从而丢弃重建所需的空间坐标；而生成式方法（如MAE）保留密集特征图以支持重建，却无法生成高层抽象。本文提出STELLAR框架，通过将视觉特征分解为语义概念与其空间分布的低秩乘积，实现解耦。该设计允许在语义令牌上执行DINO式增强对齐，同时在定位矩阵中保持精确的空间映射以支持像素级重建。实验表明，仅需16个稀疏令牌，即可在重建质量（2.60 FID）和语义性能（ImageNet准确率79.10%）上同时媲美密集骨干网络。结果表明，STELLAR是一种兼具判别与生成能力的通用稀疏表示。代码已开源。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) faces a fundamental conflict between semantic understanding and image reconstruction. High-level semantic SSL (e.g., DINO) relies on global tokens that are forced to be location-invariant for augmentation alignment, a process that inherently discards the spatial coordinates required for reconstruction. Conversely, generative SSL (e.g., MAE) preserves dense feature grids for reconstruction but fails to produce high-level abstractions. We introduce STELLAR, a framework that resolves this tension by factorizing visual features into a low-rank product of semantic concepts and their spatial distributions. This disentanglement allows us to perform DINO-style augmentation alignment on the semantic tokens while maintaining the precise spatial mapping in the localization matrix necessary for pixel-level reconstruction. We demonstrate that as few as 16 sparse tokens under this factorized form are sufficient to simultaneously support high-quality reconstruction (2.60 FID) and match the semantic performance of dense backbones (79.10% ImageNet accuracy). Our results highlight STELLAR as a versatile sparse representation that bridges the gap between discriminative and generative vision by strategically separating semantic identity from spatial geometry. Code available at https://aka.ms/stellar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。