让机器人仿真实时识别物体,速度超40帧/秒。
Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation
- 基于2D高斯点云优化渲染管线,减少冗余计算
- 采用顶K硬选择策略,实现40+帧/秒高速推理
- 适合需要实时语义分割的机器人仿真与训练
开放词汇全景重建对先进机器人和仿真至关重要。现有3D重建方法如NeRF或高斯泼溅变体常因处理高维语义特征而产生严重延迟,难以满足机器人控制环路的实时需求。本文提出Fast-SegSim,一种基于2D高斯泼溅的端到端新框架,实现高保真、3D一致且实时的开放词汇语义分割重建。核心在于优化渲染流水线,解决高通道语义特征累积的计算瓶颈:引入精确瓦片交集以降低光栅化冗余,并提出新颖的Top-K硬选择策略,利用2D高斯表示的几何稀疏性大幅简化特征累积,缓解带宽限制,实现超过40 FPS的渲染速率。Fast-SegSim在机器人应用中具有关键价值:既可作为仿真平台(如Gazebo)的高频传感器输入,其3D一致输出又能为下游感知任务提供多视角‘真实标签’。我们通过用生成标签微调目标导航中的感知模块,成功将导航成功率提升一倍。优越的渲染速度与实用价值凸显其在弥合仿真到现实差距方面的潜力。
原文摘要 · Abstract (English)
Open-vocabulary panoptic reconstruction is crucial for advanced robotics and simulation. However, existing 3D reconstruction methods, such as NeRF or Gaussian Splatting variants, often struggle to achieve the real-time inference frequency required by robotic control loops. Existing methods incur prohibitive latency when processing the high-dimensional features required for robust open-vocabulary segmentation. We propose Fast-SegSim, a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting, designed to realize real-time, high-fidelity, and 3D-consistent open-vocabulary segmentation reconstruction. Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel segmentation feature accumulation. We introduce two key optimizations: Precise Tile Intersection to reduce rasterization redundancy, and a novel Top-K Hard Selection strategy. This strategy leverages the geometric sparsity inherent in the 2D Gaussian representation to greatly simplify feature accumulation and alleviate bandwidth limitations, achieving render rates exceeding 40 FPS. Fast-SegSim provides critical value in robotic applications: it serves both as a high-frequency sensor input for simulation platforms like Gazebo, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels for fine-tuning downstream perception tasks. We demonstrate this utility by using the generated labels to fine-tune the perception module in object goal navigation, successfully doubling the navigation success rate. Our superior rendering speed and practical utility underscore Fast-SegSim's potential to bridge the sim-to-real gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。