arXiv:2604.05638cs.CV2026-04

让4D动态场景理解更智能,支持自然语言查询与跨时空推理。

PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

  • 基于多视角语义一致性机制,将2D预测融合为统一4D语义表示
  • 在动态场景语言查询任务中达到新最优性能,准确识别动作与关系
  • 适合需要跨时空语义理解的自动驾驶、机器人导航等应用

通过自然语言查询理解动态4D环境,不仅需要精确的场景重建,还需在空间、时间与视角间实现稳健的语义定位。尽管基于神经表示的近期方法已推进4D重建,但在复杂语义(如交互、时序动作、空间关系)的上下文推理方面仍受限。核心挑战在于将噪声大、视角依赖的预测转化为全局一致的4D解释。本文提出PanopticQuery框架,实现4D场景的统一查询时推理。该方法基于4D高斯点云拼贴实现高保真动态重建,并引入多视角语义共识机制,通过聚合多视角和时间帧的2D语义预测,过滤不一致输出,强化几何一致性,并借助神经场优化将2D语义提升为结构化4D语义定位。为支持评估,我们构建了新的基准数据集Panoptic-L4D,用于动态场景的语言查询。实验表明,PanopticQuery在复杂语言查询任务上达到新最佳性能,有效处理属性、动作、空间关系及多对象交互。视频演示见补充材料。

原文摘要 · Abstract (English)

Understanding dynamic 4D environments through natural language queries requires not only accurate scene reconstruction but also robust semantic grounding across space, time, and viewpoints. While recent methods using neural representations have advanced 4D reconstruction, they remain limited in contextual reasoning, especially for complex semantics such as interactions, temporal actions, and spatial relations. A key challenge lies in transforming noisy, view-dependent predictions into globally consistent 4D interpretations. We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes. Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction and introduces a multi-view semantic consensus mechanism that grounds natural language queries by aggregating 2D semantic predictions across multiple views and time frames. This process filters inconsistent outputs, enforces geometric consistency, and lifts 2D semantics into structured 4D groundings via neural field optimization. To support evaluation, we present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes. Experiments demonstrate that PanopticQuery sets a new state of the art on complex language queries, effectively handling attributes, actions, spatial relationships, and multi-object interactions. A video demonstration is available in the supplementary materials.

4D重建语义理解多视角语言查询

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。