视觉表征藏在中间层，用语言和空间对齐提取出最佳嵌入。
Perception Encoder: The best visual embeddings are not at the output of the network
- 通过语言与空间对齐，从网络中间层提取通用视觉嵌入。
- 零样本图像/视频分类与检索分别达86.6和76.9，超越现有模型。
- 适合多模态理解、密集预测任务，尤其适合大模型融合应用。
我们提出感知编码器（Perception Encoder, PE），一种基于简单视觉-语言学习训练的先进视觉编码器。传统视觉编码器依赖多种预训练目标，针对分类、描述或定位等下游任务进行优化。令人惊讶的是，经过精心调优的图像预训练流程并结合稳健的视频数据引擎后，仅使用对比式视觉-语言训练即可生成适用于各类任务的强大通用嵌入。关键在于这些嵌入位于网络中间层。为此，我们引入两种对齐方法：用于多模态语言建模的语言对齐，以及用于密集预测的空间对齐。我们的PE系列模型在多项任务上达到业界最优表现：(1) 零样本图像与视频分类及检索，同时获得86.6平均零样本ImageNet鲁棒性与76.9零样本Kinetics-400视频分类准确率；(2) 文档、图像与视频问答，实现94.6 DocVQA、80.9 InfographicVQA与82.7 PerceptionTest性能，搭配8B LLM；(3) 空间任务如检测、跟踪与深度估计，将COCO盒框mAP提升至66.0新纪录。为促进研究，我们公开了模型、代码及全新合成与人工标注视频数据集：https://github.com/facebookresearch/perception_models
原文摘要 · Abstract (English)
We introduce Perception Encoder (PE), a state-of-the-art vision encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, our PE family of models achieves best-in-class results on a wide variety of tasks, including (1) zero-shot image and video classification and retrieval, simultaneously obtaining 86.6 average zero-shot ImageNet robustness and 76.9 zero-shot Kinetics-400 video classification; (2) document, image, and video Q&A, enabling 94.6 DocVQA, 80.9 InfographicVQA, and 82.7 PerceptionTest with an 8B LLM; and (3) spatial tasks such as detection, tracking, and depth estimation, setting a new COCO state-of-the-art of 66.0 box mAP. To foster further research, we release our models, code, and novel dataset of synthetically and human-annotated videos: https://github.com/facebookresearch/perception_models
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。