评测机器人在不同光照下的语义地图生成能力，发现现有模型鲁棒性不足。
OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions
- 用大模型自动构建光照变化的评估流水线，支持动态配置
- 在模拟数据上测试主流模型，发现光照变化导致语义精度下降15%-30%
- 引入场景图评估法，揭示模型对物体关系理解的缺陷
开放语义映射（OSM）是机器人感知的核心技术，融合语义分割与SLAM。本文提出一个由大语言模型/多模态大模型驱动、可动态配置的自动化评估流水线——OSMa-Bench，用于评估OSM方案在不同室内光照条件下的表现。研究聚焦于在复杂光照变化下对先进语义映射算法的评测，构建了包含模拟RGB-D序列和真实3D重建的新型数据集，支持对映射性能的严格分析。通过对ConceptGraphs、BBQ、OpenScene等领先模型的实验，评估其在物体识别与分割中的语义保真度。同时引入场景图评估方法，分析模型对语义结构的理解能力。结果揭示了当前模型在光照变化下的脆弱性，为开发更具鲁棒性和适应性的机器人系统指明未来方向。项目主页见 https://be2rlab.github.io/OSMa-Bench/。
原文摘要 · Abstract (English)
Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ, and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Project page is available at https://be2rlab.github.io/OSMa-Bench/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。