arXiv:2603.00412cs.CV2026-03中稿 · ed

通过特征对齐增强3D视觉语言模型,保留几何语义信息

PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models

  • 用一致性损失约束点云与视觉令牌对齐,显式监督中间特征
  • 在ModelNet40上提升2.08个百分点,开放词汇任务提升7.50个百分点
  • 仅需轻量级投影器和LoRA适配器,计算开销极小

3D视觉语言模型在机器人、自动驾驶和增强现实等领域至关重要,但受限于成对3D-文本数据稀缺。现有方法仅依赖语言令牌的下一步预测损失,导致有限3D数据利用率低,中间表示中几何信息严重退化。为此,本文提出PointAlign,一种新型特征级对齐正则化方法,显式监督中间点云令牌,以在语言建模过程中保持细粒度的3D几何-语义信息。具体地,通过一致性损失约束大模型内部的点云令牌与视觉输入令牌对齐。仅训练轻量级对齐投影器和LoRA适配器,实现低计算开销的显式特征监督,有效防止几何信息退化。在ModelNet40和Objaverse数据集上的大量实验表明,该方法在分类任务上平均提升2.08个百分点,开放词汇分类任务提升7.50个百分点,3D物体描述任务(以Qwen2-72B-Instruct评估)提升4.88个百分点,验证了方法有效性。代码已公开于https://github.com/yharoldsu0627/PointAlign。

原文摘要 · Abstract (English)

The development of 3D Vision-Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-token prediction loss, using only language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable geometric information in intermediate representations. To address these limitations, we propose {\mname}, a novel feature-level alignment regularization method. {\mname} explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information throughout the language modeling process. Specifically, we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss. By training only a lightweight alignment projector and LoRA adapters, {\mname} achieves explicit feature-level supervision with minimal computational overhead, effectively preventing geometric degradation. Extensive experiments on ModelNet40 and Objaverse datasets demonstrate that our method achieves \textbf{2.08} pp improvement on average for classification tasks, with a substantial \textbf{7.50} pp gain on the challenging open-vocabulary Objaverse classification task and \textbf{4.88} pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of {\mname}. Code is publicly available at \href{https://github.com/yharoldsu0627/PointAlign}{https://github.com/yharoldsu0627/PointAlign}.

3D视觉多模态特征对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。