arXiv:2605.16832cs.CV2026-05

用彩色语义码增强点云,让大模型更准识别室内结构和家具。

Coarse Semantic Injection for LLM-Conditioned Structured Indoor Prediction

论文配图:Coarse Semantic Injection for LLM-Conditioned Structured Indoor Prediction
图 1 · 摘自论文原文
  • 将语义信息编码为红绿蓝黑四色点,与几何特征共用稀疏编码路径。
  • 在复杂场景中提升门窗定位和单个家具检测准确率,尤其在Structed3D等数据集上。
  • 仅需图像生成的语义信息,不改动原有模型结构,适合部署优化。

大型语言模型(LLM)近期被用于从3D点云标记输入中进行室内理解的结构化解码。然而,点云编码器在体素化和稀疏池化后常弱化门、窗等细长结构,并可能在杂乱场景中遗漏个别家具实例。本文提出一种保持接口的语义增强方法,将语义证据映射为粗粒度四类代码(家具、墙体、开口、其他),并编码为RGBB点接口:红色代表家具,绿色代表墙体,蓝色代表开口,黑色代表其他。其中,RGBB表示使用三个颜色通道表达四个语义状态。该语义颜色码在令牌化前附加至原始点属性,使几何与语义共享相同的稀疏令牌化路径,同时下游语言模型解码器和输出序列化过程保持不变。此外,引入轻量级路由语义偏移模块,仅在训练阶段使用辅助头进行比例/预算正则化与分析,以强化稀疏池化后的语义线索。整个流程可利用来自RGB图像的语义证据。在受控语义源条件下,各项指标在Structured3D、SpatialLM和ARKitScenes数据集上均有提升,尤其在开口定位和杂乱场景中的实例级家具检测方面表现显著。消融实验验证了语义来源、颜色编码、令牌融合及偏移注入的作用,同时表明颜色与熵的影响仍具非平凡性。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently been used as structured decoders for indoor understanding from 3D point-token inputs. However, point cloud encoders often under-represent thin structural elements such as doors and windows after voxelization and sparse pooling, and may miss individual furniture instances in cluttered scenes. We propose an interface-preserving semantic augmentation for LLM-conditioned structured decoding. The key idea is to associate semantic evidence with the point-cloud representation, reduce it to a coarse four-group code (furniture, walls, openings, and others), and encode it as an RGBB point interface: red for furniture, green for walls, blue for openings, and black for others, where RGBB denotes four semantic color states represented in three RGB channels rather than an additional fourth channel. This semantic color code is appended to the original raw point attributes before tokenization, so geometry and semantics share the same sparse tokenization path while the downstream language model decoder and output serialization remain unchanged. We further introduce a lightweight routed semantic shift module, with an auxiliary head used only for training-time ratio/budget regularization and analysis, to strengthen semantic cues after sparse pooling. The overall pipeline can use RGB-derived semantic evidence. Under these controlled semantic-source settings, the reported metrics improve across Structured3D, the SpatialLM dataset, and ARKitScenes, especially for opening localization and per-instance furniture detection in cluttered scenes. Ablations clarify the roles of semantic source, color coding, token fusion, and shift injection, while also showing that color/entropy effects remain nontrivial.

室内理解语义增强点云处理大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。