用伪任务无监督训练视觉模型，提升场景理解能力
DIP: Unsupervised Dense In-Context Post-training of Visual Representations
- 通过模拟下游任务生成伪指令，让模型在无标签数据上自适应
- 单卡A100训练不足9小时，性能超越初始模型和已有方法
- 适合需要高效改进视觉表征的工程师与研究者
我们提出DIP，一种新型无监督后训练方法，旨在增强大规模预训练视觉编码器在上下文场景理解中的密集图像表征能力。不同于依赖复杂自蒸馏架构的以往方法，DIP基于元学习思想，利用伪任务显式模拟下游上下文场景进行训练。为在无标签数据上实现后训练，我们设计了一种自动机制，结合预训练扩散模型与视觉编码器自身生成上下文任务。DIP方法简单、无监督且计算高效，仅需单张A100 GPU运行不足9小时。通过伪上下文任务学习密集表征，其在多种真实世界上下文场景理解任务中表现优异，显著优于初始编码器及已有方法，为改进密集表征提供了实用有效的解决方案。代码已公开：https://github.com/sirkosophia/DIP
原文摘要 · Abstract (English)
We introduce DIP, a novel unsupervised post-training method designed to enhance dense image representations in large-scale pretrained vision encoders for in-context scene understanding. Unlike prior approaches that rely on complex self-distillation architectures, our method trains the vision encoder using pseudo-tasks that explicitly simulate downstream in-context scenarios, inspired by meta-learning principles. To enable post-training on unlabeled data, we propose an automatic mechanism for generating in-context tasks that combines a pretrained diffusion model and the vision encoder itself. DIP is simple, unsupervised, and computationally efficient, requiring less than 9 hours on a single A100 GPU. By learning dense representations through pseudo in-context tasks, it achieves strong performance across a wide variety of downstream real-world in-context scene understanding tasks. It outperforms both the initial vision encoder and prior methods, offering a practical and effective solution for improving dense representations. Code available here: https://github.com/sirkosophia/DIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。