开源视觉模型Franca性能媲美甚至超越顶级闭源模型。
Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning
- 用嵌套俄罗斯套娃结构的多头聚类投影,渐进式细化特征聚类。
- 在多个下游任务上表现更优,尤其在去除位置偏差后提升显著。
- 全开源(数据、代码、权重),推动可复现的通用视觉模型发展。
我们提出Franca(发音为Fran-ka):首个完全开源(数据、代码、权重)的视觉基础模型,其性能可与甚至在多数情况下超越DINOv2、CLIP、SigLIPv2等顶尖闭源模型。方法基于受Web-SSL启发的透明训练流程,使用ImageNet-21K和ReLAION-2B的一个子集。针对现代自监督聚类方法中忽略语义模糊性的问题,我们引入参数高效、基于嵌套马特里什卡表示的多头聚类投影器,无需增加模型规模即可逐步细化特征聚类。此外,提出新颖的位置解耦策略,显式消除密集表示中的位置偏倚,从而改善语义编码。这在多个下游基准测试中带来一致性能提升,验证了更清洁特征空间的价值。我们的工作确立了透明、高性能视觉模型的新标准,为更可复现、可泛化的基础模型开辟路径。代码与模型检查点见https://github.com/valeoai/Franca。
原文摘要 · Abstract (English)
We present Franca (pronounced Fran-ka): free one; the first fully open-source (data, code, weights) vision foundation model that matches and in many cases surpasses the performance of state-of-the-art proprietary models, e.g., DINOv2, CLIP, SigLIPv2, etc. Our approach is grounded in a transparent training pipeline inspired by Web-SSL and uses publicly available data: ImageNet-21K and a subset of ReLAION-2B. Beyond model release, we tackle critical limitations in SSL clustering methods. While modern models rely on assigning image features to large codebooks via clustering algorithms like Sinkhorn-Knopp, they fail to account for the inherent ambiguity in clustering semantics. To address this, we introduce a parameter-efficient, multi-head clustering projector based on nested Matryoshka representations. This design progressively refines features into increasingly fine-grained clusters without increasing the model size, enabling both performance and memory efficiency. Additionally, we propose a novel positional disentanglement strategy that explicitly removes positional biases from dense representations, thereby improving the encoding of semantic content. This leads to consistent gains on several downstream benchmarks, demonstrating the utility of cleaner feature spaces. Our contributions establish a new standard for transparent, high-performance vision models and open a path toward more reproducible and generalizable foundation models for the broader AI community. The code and model checkpoints are available at https://github.com/valeoai/Franca.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。