用在线字典学习构建实时大规模3D语义地图，支持任意视觉语言模型接入。
LatentAM: Real-Time, Large-Scale Latent Gaussian Attention Mapping via Online Dictionary Learning
- 通过可学习字典与注意力机制，将高维视觉语言嵌入转为紧凑查询向量。
- 在多个数据集上实现更高特征重建保真度，且保持12-35帧/秒近实时速度。
- 无需预训练、可插拔适配不同视觉语言模型，适合开放词汇机器人感知任务。
我们提出LatentAM，一种基于在线字典学习的实时3D高斯点云映射框架，从流式RGB-D观测中构建可扩展的潜在特征图，用于开放词汇机器人感知。不同于使用特定模型解码器压缩高维视觉语言模型（VLM）嵌入的方法，LatentAM采用模型无关且无需预训练的在线字典学习策略，可在推理时灵活接入不同VLM。具体而言，每个高斯原语关联一个紧凑查询向量，通过可学习字典与注意力机制转化为近似VLM嵌入。字典由流式观测高效初始化，并在信任域正则化下在线优化，以适应场景语义演化。为支持长轨迹与大环境，我们提出基于体素哈希的高效地图管理策略：仅在GPU上优化局部活跃地图，全局地图由CPU存储与索引，从而保证有限的显存占用。在公开基准与自建大规模数据集上的实验表明，LatentAM在特征重建保真度上显著优于现有方法，同时在所测数据集上实现12-35帧/秒的近实时性能。
原文摘要 · Abstract (English)
We present LatentAM, an online 3D Gaussian Splatting (3DGS) mapping framework that builds scalable latent feature maps from streaming RGB-D observations for open-vocabulary robotic perception. Instead of distilling high-dimensional Vision-Language Model (VLM) embeddings using model-specific decoders, LatentAM proposes an online dictionary learning approach that is both model-agnostic and pretraining-free, enabling plug-and-play integration with different VLMs at test time. Specifically, our approach associates each Gaussian primitive with a compact query vector that can be converted into approximate VLM embeddings using an attention mechanism with a learnable dictionary. The dictionary is initialized efficiently from streaming observations and optimized online to adapt to evolving scene semantics under trust-region regularization. To scale to long trajectories and large environments, we further propose an efficient map management strategy based on voxel hashing, where optimization is restricted to an active local map on the GPU, while the global map is stored and indexed on the CPU to maintain bounded GPU memory usage. Experiments on public benchmarks and a large-scale custom dataset demonstrate that LatentAM attains significantly better feature reconstruction fidelity compared to state-of-the-art methods, while achieving near-real-time speed (12-35 FPS) on the evaluated datasets. Our project page, including the code, is available at: https://junwoonlee.github.io/projects/LatentAM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。