用稀疏语义高斯点实现任意视角的文本嵌入3D重建
FLEG: Feed-Forward Language Embedded Gaussian Splatting from Any Views via Compact Semantic Representation
- 通过几何-语义双分支蒸馏,支持任意多视角输入且无需相机参数
- 仅用5%的语义嵌入量,保持与密集方案相当的语义精度
- 适合需要高效存储和跨视角生成的3D内容创作场景
我们提出FLEG,一种从任意视角输入重建语言嵌入式3D高斯模型的前馈网络。现有方法受限于固定视角数,且为每个高斯点附加语言对齐语义嵌入,导致输入不灵活且存在语义冗余。为此,我们设计几何-语义双分支蒸馏框架,支持无相机参数的任意多视角输入。训练中引入新视角蒸馏策略,缓解对输入视角的过拟合。我们发现语义表示比几何表示稀疏得多,因此无需为每个高斯点附着语言嵌入。据此提出解耦语言嵌入策略,仅用一组稀疏语义高斯点表示语言信息。相比传统像素对齐的全量嵌入方案,本方法仅需5%的语义嵌入,同时保持相近的语义保真度,显著降低存储开销。大量实验表明,FLEG在重建质量与语言对齐语义表征方面均优于当前最优的前馈重建与语言嵌入高斯方法。
原文摘要 · Abstract (English)
We present FLEG, a feed-forward network that reconstructs language-embedded 3D Gaussians from arbitrary views. Previous feed-forward language-embedded Gaussian reconstruction methods are restricted to a fixed number of input views and typically attach a language-aligned semantic embedding to each Gaussian, resulting in impractical input settings and semantic redundancy. In contrast, we introduce a geometry-semantic dual-branch distillation framework that enables flexible input from arbitrary multi-view images without camera parameters. We also propose a novel-view-based distillation strategy during training that mitigates overfitting to input views. In addition, we observe that semantic representations are significantly sparser than geometric ones, and per-Gaussian language embedding is unnecessary. To exploit this sparsity, we design a decoupled language embedding strategy that represents language information with a sparse set of semantic Gaussians, rather than attaching embeddings to every Gaussian. Compared with dense pixel-aligned per-Gaussian embedding schemes, our method uses only 5\% of the language embeddings while maintaining comparable semantic fidelity, effectively reducing storage costs. Extensive experiments demonstrate that FLEG outperforms state-of-the-art feed-forward reconstruction and language-embedded Gaussian methods in both reconstruction quality and language-aligned semantic representation. Project page: https://fangzhou2000.github.io/projects/fleg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。