arXiv:2607.00514cs.CVcs.AI2026-07被引 1

用2D模型指导4D点云学习,实现细粒度跨模态对齐。

Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

论文配图:Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
图 1 · 摘自论文原文
  • 通过3D点与2D图像块的密集对应关系,实现跨模态特征蒸馏。
  • 在4个基准上均优于现有方法,小模型性能媲美大模型。
  • 适合需要高效标注和跨域迁移的动态点云任务。

动态4D点云(由深度传感器和激光雷达随时间捕捉的3D点序列)的理解是机器人与具身感知的核心。但密集标注成本高昂,自监督预训练成为获取可迁移表征的自然路径。现有方法多为单模态预训练,少数基于2D基础模型的方法仅使用每段视频的全局嵌入,忽略了模型计算的局部语义信息。为此,我们提出Cross4D-JEPA,一种教师-学生框架,将冻结的2D图像模型DINOv2或视频模型V-JEPA 2的知识蒸馏至4D点云编码器。该方法结合:(1) 每个3D点与其投影对应的2D图像块特征之间的密集跨模态对应;(2) 无掩码、无负样本、无解码器的逐点目标函数。我们在MSR-Action3D、DeformingThings4D、NTU-RGB+D 60和HOI4D四个基准上评估,结果表明,在相同协议下,所提方法在四基准上一致优于单模态及全局跨模态基线,并与更重的已有4D方法相当;进一步分析显示,性能提升主要源于对应关系的粒度而非教师模态。此外,该方法不仅提升识别精度,其稠密表征还能跨域迁移,提高标签效率,并在相同训练预算下改善全标注微调效果,且13倍小的编码器即可匹敌重型池化主干。

原文摘要 · Abstract (English)

Automatic understanding of dynamic 4D point clouds, the 3D-point sequences captured over time by depth sensors and LiDAR, is central to robotics and embodied perception. Yet annotating them densely is expensive, making self-supervised pretraining the natural route to transferable representations. Existing pretext tasks, however, are almost entirely intra-modal, and the few methods that transfer knowledge from 2D foundation models rely on a single global embedding per clip, discarding the rich per-patch semantics that these models compute. To address this gap, we propose Cross4D-JEPA, a teacher-student method that distills a frozen 2D foundation model, an image model DINOv2, or a video model V-JEPA 2, into a 4D point encoder. The proposed method combines (1) a dense cross-modal correspondence that maps every 3D point to the teacher patch feature it projects to, and (2) a per-point objective that trains the student to match these features in latent space with no masking, negatives, or decoder. We evaluate Cross4D-JEPA on four benchmarks, MSR-Action3D, DeformingThings4D, NTU-RGB+D 60, and HOI4D, against intra-modal and global cross-modal baselines. Experimental results show that, under a matched protocol, the proposed method consistently outperforms intra-modal and global cross-modal baselines across the four benchmarks and is competitive with heavier published 4D methods; further analysis attributes this gain primarily to the granularity of the correspondence rather than the teacher modality. Beyond recognition accuracy, the dense representation learned by Cross4D-JEPA transfers across domains, improves label efficiency, and improves full-label fine-tuning under the same training budget, while a 13x smaller encoder matches a heavyweight pooling backbone.

4D点云跨模态自监督知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。