用程序生成数据训练模型，实现知识与感知分离，零样本表现优异
Separating Knowledge and Perception with Procedural Data
- 仅用程序生成数据训练，通过视觉记忆库直接应用到各类任务
- 在细粒度分类上比真实数据模型高8%~15%，图像相似性误差小于1%
- 适合研究模型泛化能力或追求零样本部署的开发者
我们仅使用程序生成数据训练表示模型，并通过显式的参考图像嵌入数据库——视觉记忆，在无需额外训练的情况下应用于视觉相似性、分类和语义分割任务。与以往视觉记忆方法不同，本方法实现了对所有现实图像的完全隔离，同时保持强性能。相比在Places数据集上训练的模型，该程序生成模型在NIGHTS视觉相似性任务上表现相差不足1%，在CUB200和Flowers102细粒度分类上分别领先8%和15%，在ImageNet-1K分类上误差在10%以内。其零样本分割表现也强劲，在COCO上的R²值仅比真实数据训练模型低10%。最后，我们分析了程序生成与真实数据模型的差异，发现同一物体的不同部分在程序模型中表示差异大，导致记忆搜索错误，解释了性能差距的成因。
原文摘要 · Abstract (English)
We train representation models with procedural data only, and apply them on visual similarity, classification, and semantic segmentation tasks without further training by using visual memory -- an explicit database of reference image embeddings. Unlike prior work on visual memory, our approach achieves full compartmentalization with respect to all real-world images while retaining strong performance. Compared to a model trained on Places, our procedural model performs within $1\%$ on NIGHTS visual similarity, outperforms by $8\%$ and $15\%$ on CUB200 and Flowers102 fine-grained classification, and is within $10\%$ on ImageNet-1K classification. It also demonstrates strong zero-shot segmentation, achieving an $R^2$ on COCO within $10\%$ of the models trained on real data. Finally, we analyze procedural versus real data models, showing that parts of the same object have dissimilar representations in procedural models, resulting in incorrect searches in memory and explaining the remaining performance gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。