arXiv:2511.13211cs.CV2025-11AAAI被引 13

解决大规模3D与文本精细对齐难题,提升跨模态检索精度。

3DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scale

  • 提出动态注意力策略,实现细粒度文本-点云对应建模。
  • 在200万对数据上达到领先性能,检索准确率显著优于传统方法。
  • 适合需要高精度3D-文本匹配的研究者与工业应用开发者。

尽管3D-text跨模态对齐取得进展,现有方法仍难以将细粒度文本语义与详细几何结构对齐,且在大规模3D数据库上性能明显下降。为此,我们提出3DAlign-DAER框架,通过动态注意力策略(DAP)与高效检索策略(ERS),实现多样跨模态任务中的精准对齐。训练阶段,动态注意力策略利用分层注意力融合模块(HAF)建立可学习的细粒度词元-点注意力,并通过蒙特卡洛树搜索结合混合奖励信号动态校准注意力权重,增强文本描述与局部几何的对齐。推理阶段,高效检索策略在大规模嵌入空间中进行层次化搜索,相比KNN等传统方法在准确率与效率上均有提升。为支持研究并训练模型,我们构建了包含200万对文本-3D样本的Align3D-2M数据集,提供充分的细粒度标注。大量实验表明,3DAlign-DAER在多个基准上表现优异。代码、模型及数据集将公开发布。

原文摘要 · Abstract (English)

Despite recent advancements in 3D-text cross-modal alignment, existing state-of-the-art methods still struggle to align fine-grained textual semantics with detailed geometric structures, and their alignment performance degrades significantly when scaling to large-scale 3D databases. To overcome this limitation, we introduce 3DAlign-DAER, a unified framework designed to align text and 3D geometry via the proposed dynamic attention policy and the efficient retrieval strategy, capturing subtle correspondences for diverse cross-modal retrieval and classification tasks. Specifically, during the training, our proposed dynamic attention policy (DAP) employs the Hierarchical Attention Fusion (HAF) module to represent the alignment as learnable fine-grained token-to-point attentions. To optimize these attentions across different tasks and geometric hierarchies, our DAP further exploits the Monte Carlo tree search to dynamically calibrate HAF attention weights via a hybrid reward signal and further enhances the alignment between textual descriptions and local 3D geometry. During the inference, our 3DAlign-DAER introduces an Efficient Retrieval Strategy (ERS) to leverage efficient hierarchical searching in the large-scale embedding spaces, outperforming traditional methods (e.g., KNN) in accuracy and efficiency. Furthermore, to facilitate text-3D alignment research and train our 3DAlign-DAER, we construct Align3D-2M, a large-scale dataset featuring 2M text-3D pairs, to provide sufficient fine-grained cross-modal annotations. Extensive and comprehensive experiments demonstrate the superior performance of our 3DAlign-DAER on diverse benchmarks. We will release our codes, models, and datasets. Our code and updates are available at https://github.com/waltstephen/Cost-Effective-Communication.

3D对齐跨模态动态注意力高效检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。