让文字指令自动找到3D场景中最佳拍摄角度。
CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels
- 用语言生成几何伪标签，指导相机位姿优化。
- 在38个真实场景上，效果优于传统搜索和轨迹生成方法。
- 适合需要精准视觉布局的AI摄影、虚拟拍摄场景。
3D高斯溅射（3DGS）实现了逼真的实时新视角合成，但确定理想的虚拟相机位置仍依赖人工操作。现有语言引导方法多聚焦于目标定位，难以控制单帧画面中的主体朝向或构图布局。为此，我们提出新任务——文本指令视角定位（TIVG），旨在寻找一个6自由度相机位姿，使渲染画面与文本指令一致。我们提出CapFrame框架，采用可部分微分的流程：通过多模态大模型检索并评估相关视图，将指令转化为朝向与布局伪标签，并在3DGS中利用可微优化结合布局与朝向损失进行位姿精调。在38个真实场景、135条指令上的实验表明，该方法在视觉语言模型指标、多模态大模型评价及用户研究中均优于启发式搜索与适配轨迹生成基线。代码已开源：https://github.com/jirongli/CapFrame
原文摘要 · Abstract (English)
3D Gaussian Splatting (3DGS) enables photorealistic real-time novel view synthesis, yet placing a virtual camera to capture a desired frame remains largely manual. Existing language-guided approaches in 3D scenes mainly focus on object-centric grounding, determining what to observe but rarely controlling how it should appear in a single frame, such as subject orientation or frame layout. To address this limitation, we introduce a new task, Text-Instructed Viewpoint Grounding (TIVG), which aims to identify a 6-DoF camera pose in a 3D Gaussian scene whose rendered frame aligns with a text instruction. To solve this task, we propose CapFrame, a partially differentiable framework that converts language into geometric pseudo labels for camera pose optimization. CapFrame follows a Retrieve-Translate-Refine pipeline: it retrieves relevant views and ranks them through a Question-Evaluation process with MLLMs, translates the instruction into orientation and layout pseudo labels, and refines the camera pose via differentiable optimization with layout and orientation losses in 3DGS. Experiments on 38 real-world scenes with 135 instructions indicate that CapFrame produces viewpoints better aligned with texts than heuristic viewpoint search and adapted trajectory generation baselines, validated by VLM metrics, MLLM judges, and user studies. Code is available at: https://github.com/jirongli/CapFrame
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。