用跨模型一致性选精炼图像表征,不训练也能提效
Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis
- 通过双模型CCA找共享语义,线性投影实现表征筛选
- 维度降低75%以上,下游任务准确率最高提升12.6%
- 适合想优化预训练表征的开发者和研究者
现代视觉系统广泛使用预训练图像编码器的表征,但这些表征常冗余且模型特异。本文提出一种无需训练的后处理方法:基于两个预训练编码器输出的共现结构,通过典型相关分析(CCA)找到线性投影,实现表征选择与降维。该方法保留共享语义内容,剔除冗余维度,相比仅在单一空间操作的PCA,利用跨模型一致性指导表征提炼。实验在ImageNet-1k、CIFAR-100、MNIST等数据集上显示,表征可压缩超75%,同时下游性能显著提升,最高达12.6%准确率增益;也可在固定维度下通过迁移更大或微调模型的表征增强性能。
原文摘要 · Abstract (English)
Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。