用嵌套网格壳实现可微渲染,支持真实世界大场景重建。
Meshtryoshka: Differentiable Rendering of Real-World Scenes via Mesh Rasterization

- 通过等值面提取构建嵌套网格壳,结合符号距离函数动态生成几何。
- 在真实大场景上实现高质量新视角合成,接近非网格方法性能。
- 兼容通用渲染器,无需对顶点位置做可微处理,降低实现门槛。
可微渲染已成为3D重建与新视角合成的强大工具。当前先进方法多采用定制化的3D几何与外观表示及专用渲染器,但计算机图形学下游任务普遍依赖3D网格。尽管已有工作尝试基于网格的可微渲染,但局限于物体中心场景,难以重建大规模无界场景。本文提出Meshtryoshka,一种新颖的网格可微渲染框架,结合现成三角形光栅化器与嵌套网格壳结构(类似俄罗斯套娃)。每轮前向传播中,网格壳通过符号距离函数的等值面提取生成,顶点不透明度由符号距离决定。各网格壳独立光栅化后,通过Alpha混合合成最终图像。关键在于顶点位置仅通过不透明度梯度反向传播至符号距离函数间接更新,因此兼容无需对顶点可微的现成网格渲染器。在物体中心场景中,性能媲美基于表面的可微渲染方法;在无界真实场景中,仍能实现高质量新视角合成,逼近最先进非网格方法水平。结果表明,仅使用常规计算机图形学工具即可解决可微渲染问题。
原文摘要 · Abstract (English)
Differentiable rendering has emerged as a powerful approach for 3D reconstruction and novel view synthesis. State-of-the-art differentiable rendering methods combine a variety of custom representations of 3D geometry and appearance with specialized renderers. However, most downstream tasks in computer graphics rely on 3D meshes. While prior work has attempted differentiable rendering with mesh representations, these approaches are limited to object-centric scenes and fail to reconstruct large-scale, unbounded scenes. In this work, we introduce Meshtryoshka, a novel mesh differentiable rendering framework that combines an off-the-shelf triangle rasterizer with a 3D representation that consists of nested mesh shells which resemble a matryoshka doll. In every forward pass, the mesh shells are extracted anew from a 3D signed distance function via iso-surface extraction, and the opacities for each vertex are computed as a function of signed distance. Each mesh shell is then rasterized independently, and the final image is created via alpha compositing. Crucially, mesh vertex positions are updated only indirectly via gradients that flow through the opacity values into the signed distance function, and hence, our method is compatible with off-the-shelf mesh renderers that need not be differentiable with respect to vertex positions. On object-centric scenes, our method performs competitively with surface-based differentiable rendering techniques. Our differentiable mesh rendering method scales to unbounded, real-world 3D scenes, where it yields high-quality novel view synthesis results approaching those of state-of-the-art, non-mesh methods. Our method suggests that it may be possible to solve the differentiable rendering problem without relying on specialized renderers, only using conventional tools from the computer graphics toolbox.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。