用2D图像训练3D模型,速度快且省资源。
CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Images
- 用图像和点云配对学习,不依赖掩码,简化结构。
- 在ModelNet40上达94.2%准确率,仅需1410万参数。
- 适合资源受限场景,单卡6小时即可完成预训练。
图像到点云的跨模态学习旨在缓解3D数据集规模不足的问题。现有方法常导致模型庞大、训练慢,难以部署于资源受限环境。本文提出CrossJEPA,一种基于联合嵌入预测架构的跨模态方法,利用图像基础模型知识,训练预测器从对应3D点云中推断特定渲染2D视图的嵌入,引入无需掩码的JEPA式预训练策略。通过跨域投影信息条件化,净化目标域专属语义干扰。采用冻结教师与一次性目标嵌入缓存机制,实现计算效率提升。CrossJEPA在合成数据集ModelNet40线性探测达94.2%,真实世界数据集ScanObjectNN达88.3%,仅使用14.1M预训练参数(其中点编码器8.5M),单张标准GPU上约6小时完成训练。结果表明其兼具高性能、低内存与快速训练优势,适用于知识蒸馏驱动的3D表示学习。我们从直观、理论和实证角度分析设计,并进行充分消融实验。代码将公开。
原文摘要 · Abstract (English)
Image-to-point cross-modal learning has emerged to address the scarcity of large-scale 3D datasets in 3D representation learning. However, current methods that leverage 2D data often result in large, slow-to-train models, making them computationally expensive and difficult to deploy in resource-constrained environments. The architecture design of such models is therefore critical, determining their performance, memory footprint, and compute efficiency. The Joint-embedding Predictive Architecture (JEPA) has gained wide popularity in self-supervised learning for its simplicity and efficiency, but has been under-explored in cross-modal settings, partly due to the misconception that masking is intrinsic to JEPA. In this light, we propose CrossJEPA, a simple Cross-modal Joint Embedding Predictive Architecture that harnesses the knowledge of an image foundation model and trains a predictor to infer embeddings of specific rendered 2D views from corresponding 3D point clouds, thereby introducing a JEPA-style pretraining strategy beyond masking. By conditioning the predictor on cross-domain projection information, CrossJEPA purifies the supervision signal from semantics exclusive to the target domain. We further exploit the frozen teacher design with a one-time target embedding caching mechanism, yielding amortized efficiency. CrossJEPA achieves a new state-of-the-art in linear probing on the synthetic ModelNet40 (94.2%) and the real-world ScanObjectNN (88.3%) benchmarks, using only 14.1M pretraining parameters (8.5M in the point encoder), and about 6 pretraining hours on a standard single GPU. These results position CrossJEPA as a performant, memory-efficient, and fast-to-train framework for 3D representation learning via knowledge distillation. We analyze CrossJEPA intuitively, theoretically, and empirically, and extensively ablate our design choices. Code will be made available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。