用视觉语言模型直接从单图生成可编辑的Blender代码场景
Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models
- 分阶段重构几何、材质、光照等要素，直接在Blender代码空间中操作
- 相比端到端方法，重建精度显著提升，像素级与语义保真度更高
- 适合需要3D可编辑场景的创意设计、影视制作等应用
逆图形学是一个长期存在且高度不明确的问题，目标是将图像还原为可编辑的3D场景，支持重渲染、重打光和交互操作。本文研究预训练视觉语言模型（VLM）能否仅通过单张图像直接执行可执行逆图形，将场景重建为可编辑的Blender程序，无需专用2D/3D基础模型、可微分渲染或多视角监督。我们提出分阶段可执行逆图形（SEIG）框架，通过逐步优化几何、材质、构图和光照等场景因素，直接在可执行Blender代码空间中完成重建。我们在多种场景上评估该框架，使用涵盖像素级、感知和语义保真度的多种指标。实验表明，分阶段重建显著提升了重建质量，凸显了任务分解对通用VLM实现可执行逆图形的重要性。最后，我们展示了由可编辑的Blender场景支持的多种下游应用。
原文摘要 · Abstract (English)
Inverse graphics is a longstanding and highly underconstrained problem that seeks to reconstruct images as editable 3D scenes which can be rendered, relit, and manipulated. In this work, we investigate whether pretrained vision-language models (VLMs) can perform executable inverse graphics directly from a single image by reconstructing a scene as an editable Blender program, without relying on specialized 2D or 3D foundation models, differentiable rendering, or multi-view supervision. We introduce Staged Executable Inverse Graphics (SEIG), an agentic framework that reconstructs a 3D scene from a single image by progressively refining scene factors including geometry, materials, composition, and lighting directly in executable Blender code space. We evaluate our framework across diverse scenes using a range of reconstruction metrics spanning pixel-level, perceptual, and semantic fidelity. Our experiments show that staged reconstruction substantially improves reconstruction fidelity, highlighting the importance of task decomposition for executable inverse graphics with general-purpose VLMs. Finally, we showcase various downstream applications enabled by the reconstructed editable Blender scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。