将点云视为序列,用因果预测实现无解码器的自监督学习。
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

- 把点云切片成几何序列,用因果Transformer预测下一个点。
- 在ScanObjectNN等数据集上性能提升0.5%~1.1%,最高达93.8%。
- 无需重建或显式几何生成,适合做通用3D基础模型预训练。
随着多模态大模型和预测预训练的发展,如何为3D点云设计更契合下一词与下一嵌入学习的预训练范式成为关键问题。现有自监督方法主要依赖掩码重建或显式几何生成,仍局限于输入恢复而非预测依赖建模。本文提出PointNTP,将点云预训练重构为完全因果、无需解码器的潜在下一词预测任务。具体地,将点云按局部块中心几何结构分割并序列化为3D token序列,采用前缀仅条件因果Transformer建模,并通过带停止梯度目标的移位预测目标进行训练。该设计使模型直接在潜在空间中学习结构依赖,无需重建解码器或显式几何恢复。大量实验表明,所提方法在多个下游任务中表现优异:在ScanObjectNN的OBJ_BG、OBJ_ONLY和PB_T50_RS上分别达到93.8%(+0.5%)、92.6%(+0.3%)和89.3%(+1.1%);在ShapeNetPart上获得85.0%(+0.1%)的Cls.mIoU;在S3DIS Area 5上达71.1% mAcc。整体来看,无解码器的因果潜在预测提供了一种简单、可扩展且可能跨模态通用的点云自监督学习范式,为3D数据的基础模型预测学习提供了新视角。
原文摘要 · Abstract (English)
With the rapid progress of multimodal foundation models and predictive pre-training, an important open question is how to equip 3D point clouds with a pre-training paradigm that is better aligned with next-token and next-embedding learning. Existing point-cloud self-supervised methods are largely built on masked reconstruction or explicit geometric generation, and thus remain tied to input recovery rather than predictive dependency modeling. In this paper, we introduce PointNTP, which reformulates point cloud pre-training as a fully causal, decoder-free latent Next-Token Prediction problem. Specifically, each point cloud is first partitioned into local patches and serialized into a structured 3D token sequence according to patch-center geometry. The resulting sequence is then modeled by a causal Transformer under prefix-only conditioning, and trained with a shift-based prediction objective stabilized by stop-gradient targets. This design enables the model to learn structural dependencies directly in latent space, without reconstruction decoders or explicit geometric recovery. Extensive experiments demonstrate that the proposed PointNTP is highly competitive across multiple downstream tasks: it achieves 93.8%(+0.5%), 92.6%(+0.3%), and 89.3%(+1.1%) on OBJ_BG, OBJ_ONLY, and PB_T50_RS of ScanObjectNN, respectively; obtains 85.0%(+0.1%) in Cls.mIoU on ShapeNetPart; and reaches 71.1% mAcc on S3DIS Area 5. Overall, decoder-free causal latent prediction provides a simple, scalable, and potentially modality-agnostic paradigm for point-cloud self-supervised learning, offering a new 3D perspective on foundation-style predictive learning for 3D data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。