用解码器单架构实现更高效高精度的视图合成
DVSM: Decoder-only View Synthesis Model Done Right
- 采用仅解码器结构，隐式存储场景为键值缓存
- 参数更少但性能超越主流编码器-解码器模型
- 适合追求高效高保真视图生成的研究与应用
近期大型视图合成模型（LVSM）普遍采用编码器-解码器架构，将重建与渲染分属不同网络。我们重新审视这一设计。通过受控实验发现，仅使用解码器的架构——将场景隐式表示为键值缓存——在相同渲染复杂度下，以更少参数实现了更优性能。进一步分析表明，将颜色输入重建网络与仅相机输入的渲染网络共享权重，可使同一视角下的特征对齐更佳，从而提升图像合成效果。基于此，我们提出名为DVSM的模型，融合基础模型先验与分阶段补丁尺寸策略，实现更好的效率-质量平衡。结果在多个基准上达到新最佳性能，部分情况下甚至优于密集输入视角下的每场景优化3DGS。
原文摘要 · Abstract (English)
Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks. We re-examine this design. Through controlled experiments, we show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity. Further analysis shows that sharing weights between the color-input reconstruction network and the camera-only rendering network better aligns their features at the same viewpoint, facilitating image synthesis. Building on this finding, our model, dubbed DVSM, further incorporates foundation model priors and stage-wise patch sizing for an improved efficiency-quality tradeoff. Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。