arXiv:2509.00374cs.CV2025-09TPAMI被引 1

用点云特征直接适配多模态模型,高效实现3D点云分析。

Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis

  • 通过点嵌入与动态提示生成,直接处理点云数据
  • 仅用少量参数即可适配任意预训练模型,提升3D分析性能
  • 适合想快速部署多模态3D模型的研究者和工程师

一维文本与二维视觉领域的参数高效微调策略已表现出显著效果。然而,由于点云数据稀缺,大规模3D模型预训练仍具挑战性。现有通过“高维到低维”映射将预训练视觉模型应用于3D领域的方法,常导致空间几何信息丢失,且缺乏通用适配框架。本文提出自适应点提示微调(APPT),直接利用点特征校准任意模态的异构基础模型以进行3D点云分析。方法将原始点云通过局部几何聚合生成点嵌入,并经线性层确保与冻结预训练模型无缝对接;针对点云固有的无序性,采用置换不变特征捕获点嵌入相对位置,增强自注意力机制中的位置信息;进一步设计共享权重的提示生成器,动态生成点提示,不增加额外参数,将其拼接至冻结模型,提供丰富的全局结构信息,弥补异构数据中结构上下文的缺失。

原文摘要 · Abstract (English)

Parameter-efficient fine-tuning strategies for foundation models in 1D textual and 2D visual analysis have demonstrated remarkable efficacy. However, due to the scarcity of point cloud data, pre-training large 3D models remains a challenging task. While many efforts have been made to apply pre-trained visual models to 3D domains through "high-to-low" mapping, these approaches often lead to the loss of spatial geometries and lack a generalizable framework for adapting any modality to 3D. This paper, therefore, attempts to directly leverage point features to calibrate the heterogeneous foundation model of any modality for 3D point cloud analysis. Specifically, we propose the Adaptive Point-Prompt Tuning (APPT) method, which fine-tunes pre-trained models with a modest number of parameters, enabling direct point cloud processing without heterogeneous mappings. We convert raw point clouds into point embeddings by aggregating local geometry to capture spatial features followed by linear layers to ensure seamless utilization of frozen pre-trained models. Given the inherent disorder of point clouds, in contrast to the structured nature of images and language, we employ a permutation-invariant feature to capture the relative positions of point embeddings, thereby obtaining point tokens enriched with location information to optimize self-attention mechanisms. To calibrate self-attention across source domains of any modality to 3D and reduce computational overhead, we introduce a prompt generator that shares weights with the point embedding module, dynamically producing point-prompts without adding additional parameters. These prompts are then concatenated into a frozen foundation model, providing rich global structural information and compensating for the lack of structural context in the heterogeneous data.

3D点云提示微调多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。