用可学习的2D高斯表示图像，实现与传统方法相当的识别性能。
GViT: Representing Images as Gaussians for Visual Recognition
- 将图像转为数百个可学习的2D高斯分布，联合优化位置与颜色等参数。
- 在ImageNet-1k上达到76.9%的准确率，接近传统ViT性能。
- 适合对视觉表征机制感兴趣的研究者，尤其关注非网格输入设计者。
我们提出GViT，一种分类框架，摒弃传统的像素或补丁网格输入表示，转而采用一组紧凑的可学习2D高斯分布。每张图像被编码为数百个高斯，其位置、尺度、方向、颜色和透明度与顶部的ViT分类器共同优化。利用分类器梯度作为建设性引导，推动高斯向类别显著区域集中，同时通过可微渲染器优化图像重建损失。实验表明，结合2D高斯输入表示与我们的GViT引导机制，使用标准的ViT架构即可在ImageNet-1k上达到76.9%的Top-1准确率，接近传统基于补丁的ViT性能。
原文摘要 · Abstract (English)
We introduce GVIT, a classification framework that abandons conventional pixel or patch grid input representations in favor of a compact set of learnable 2D Gaussians. Each image is encoded as a few hundred Gaussians whose positions, scales, orientations, colors, and opacities are optimized jointly with a ViT classifier trained on top of these representations. We reuse the classifier gradients as constructive guidance, steering the Gaussians toward class-salient regions while a differentiable renderer optimizes an image reconstruction loss. We demonstrate that by 2D Gaussian input representations coupled with our GVIT guidance, using a relatively standard ViT architecture, closely matches the performance of a traditional patch-based ViT, reaching a 76.9% top-1 accuracy on Imagenet-1k using a ViT-B architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。