让视觉模型与图文模型对齐，保留几何细节同时提升零样本识别能力
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
- 将视觉模型特征视为'视觉语言'，通过正交映射对齐图文模型语义空间
- 在多个基准上实现零样本图像识别与分割性能显著提升，开销极低
- 无需标签或模型更新，适用于任意预训练视觉模型的通用对齐
基础模型推动了计算机视觉的快速发展，但主流的图文基础模型（VLMs）与纯视觉基础模型（VFMs）之间仍存在部分不兼容。VLMs具备语言引导的语义对齐能力，但视觉表征较粗糙；而VFMs能学习精细的感知几何特征，却缺乏语义标注。我们提出GPUA（Geometry-Preserving Unsupervised Alignment），一种融合两者互补优势的框架。受跨语言对齐启发，GPUA将VFM特征视为一种‘视觉语言’，学习一个正交映射，将VFM空间转换到VLM的语义空间，同时保持几何结构，缩小模态差距，且无需标签或模型参数更新。GPUA为任务无关设计，仅需访问预训练模型的特征。在多个基准上的实验表明，该方法显著提升了跨模型兼容性，并在下游零样本识别与分割任务中取得明显增益，计算开销极小。代码已开源：https://github.com/Yuteam14/GPUA
原文摘要 · Abstract (English)
Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。