提升视图合成速度与效果，用新结构让模型更高效
Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention
- 分路设计分离输入与目标视图处理，减少冗余计算
- 2个输入视图下达29.86 dB PSNR，训练快2倍、推理快4.4倍
- 适合需要快速生成或未知视图数场景的视觉应用
基于Transformer的前馈视图合成模型如LVSM通过全自注意力机制实现多视图建模。本文指出其全自注意力结构存在二次复杂度问题，且对异构特征共享僵化。提出Efficient-LVSM，采用双流架构，对输入视图使用内部自注意力，对目标视图采用自注意力后交叉注意力，避免无效计算。在RealEstate10K数据集上，2个输入视图下达到29.86 dB PSNR，较LVSM提升0.2 dB，训练收敛速度提升2倍，推理速度提升4.4倍。该模型在多个基准上表现领先，具备强零样本泛化能力，支持基于键值缓存的增量推理，得益于其解耦设计。
原文摘要 · Abstract (English)
Feedforward models for novel view synthesis (NVS) have recently advanced by transformer-based methods like LVSM, using attention among all input and target views. In this work, we argue that its full self-attention design is suboptimal, suffering from quadratic complexity with respect to the number of input views and rigid parameter sharing among heterogeneous tokens. We propose Efficient-LVSM, a dual-stream architecture that avoids these issues with a decoupled co-refinement mechanism. It applies intra-view self-attention for input views and self-then-cross attention for target views, eliminating unnecessary computation. Efficient-LVSM achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed. Efficient-LVSM achieves state-of-the-art performance on multiple benchmarks, exhibits strong zero-shot generalization to unseen view counts, and enables incremental inference with KV-cache, thanks to its decoupled designs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。