用2D高斯点阵压缩图像,实现高效视觉语言对齐。
GaussianVision: Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting
- 用2D高斯点阵替代像素,构建紧凑自适应图像表示。
- 输入压缩3-23.5倍,零样本性能媲美传统方法,训练参数仅9.7%-13.8%。
- 适合边缘计算与低带宽场景下的多模态学习,提升传输效率。
当前视觉语言模型依赖海量图文数据训练的RGB编码器,但其在像素域存在两大结构缺陷:(i) 边缘设备向云端传输密集RGB图像能耗高、成本大;(ii) 基于块的分词使序列过长,超出注意力机制上下文限制。本文探索2D高斯点阵(2DGS)作为对齐新基底——一种由彩色非对称高斯函数组成的紧凑、空间自适应表示。我们构建了可扩展的2DGS流水线,采用结构化初始化、亮度感知剪枝和批处理CUDA内核,实现超90倍加速与约97%的GPU利用率。进一步通过重用冻结的基于RGB的Transformer主干,结合轻量级点阵感知输入分支与每通道感知重构器,将对比语言图像预训练(CLIP)适配至2DGS,仅训练总参数的9.7%至13.8%。在包含1280万条数据的DataComp数据集上训练后,2DGS编码器在CLIP基准的38个数据集上达到可比零样本性能,同时实现3至23.5倍的输入压缩。结果表明2DGS是可行的多模态基底,揭示架构瓶颈,并为兼具语义能力与传输效率的边缘-云学习开辟路径。
原文摘要 · Abstract (English)
Modern vision language pipelines are driven by RGB vision encoders trained on massive image text corpora. While these pipelines have enabled impressive zero-shot capabilities and strong transfer across tasks, they still inherit two structural inefficiencies from the pixel domain: (i) transmitting dense RGB images from edge devices to the cloud is energy-intensive and costly, and (ii) patch-based tokenization explodes sequence length, stressing attention budgets and context limits. We explore 2D Gaussian Splatting (2DGS) as an alternative visual substrate for alignment: a compact, spatially adaptive representation that parameterizes images by a set of colored anisotropic Gaussians. We develop a scalable 2DGS pipeline with structured initialization, luminance-aware pruning, and batched CUDA kernels, achieving over 90x faster fitting and about 97% GPU utilization compared to prior implementations. We further adapt contrastive language-image pre-training (CLIP) to 2DGS by reusing a frozen RGB-based transformer backbone with a lightweight splat-aware input stem and a perceiver resampler, training only 9.7% to 13.8% of the total parameters. On a 12.8M dataset from DataComp, GS encoders yield competitive zero-shot performance on 38 datasets from the CLIP benchmark while compressing inputs 3x to 23.5x relative to pixels. Our results establish 2DGS as a viable multimodal substrate, pinpoint architectural bottlenecks, and open a path toward representations that are both semantically powerful and transmission-efficient for edge-cloud learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。