无需优化即可快速生成视频高斯表示,支持跨视频泛化与高分辨率渲染。
HyperGS: Fast and Generalizable Gaussian Video Representation

- 通过可学习查询的时空变换器直接预测每帧高斯参数,实现单次前向传播生成。
- 编码速度提升10^4至10^5倍,零样本泛化至720p视频,重建质量相当。
- 设计动态几何正则化缓解训练崩溃,适合追求高效通用视频建模的研究者。
高斯点阵已成为有效的视频表示方法,但现有方法依赖逐视频优化,导致编码缓慢且泛化能力差。为此,我们提出HyperGS,一种前馈式、无需优化的方法,可在单次前向传播中直接从任意视频生成高斯表示,使编码与解码速度提升数个数量级,同时在更高分辨率下实现对分布外视频的零样本泛化。HyperGS采用分解式时空变换器提取视频特征,再通过可学习查询的变换器为每帧生成8参数高斯表示。我们发现,直接在多样化视频上预测高斯会引发尖锐退化导致训练崩溃,因此引入动态适应的秩基几何正则化以稳定优化。HyperGS在匹配重建质量下实现10^4–10^5倍的编码加速,并在未重新编码的情况下支持720p视频零样本泛化。在K400、SSv2和UCF101数据集上,相比先前视频编码器,以更小的表示尺寸提升PSNR达+2.9–3.1 dB。通过单次前向传播预测显式2D高斯,HyperGS结合了高斯点阵的快速灵活渲染与前馈预测的速度与泛化优势,推动高斯表示成为快速、通用视频建模的实际方向。
原文摘要 · Abstract (English)
Gaussian Splatting has emerged as an effective representation for video, but existing methods rely on per-video optimization. This leads to slow encoding and limits generalization across videos. To amortize this optimization, we propose HyperGS, a feedforward, optimization-free approach that directly predicts Gaussian representations from any video in a single forward pass, speeding up encoding and decoding by orders of magnitude while generalizing to out-of-distribution videos at higher resolutions. In HyperGS, we design a factorized spatiotemporal Transformer to extract tokens from video, and a learnable query-based Transformer to obtain 8-parameter Gaussian representations for each video frame. We find that naively predicting Gaussians across diverse videos induces a needle-like degeneration that collapses training, and address this with a rank-based geometric regularizer whose strength adapts dynamically to stabilize optimization. HyperGS achieves encoding at $10^4$--$10^5\times$ the speed of per-video Gaussian optimization at matched reconstruction quality while generalizing zero-shot to $720p$ video, enabling higher-resolution rendering without re-encoding. HyperGS improves PSNR by +2.9--3.1 dB over the prior video encoders on K400, SSv2, and UCF101 at a smaller video representation size. By predicting explicit 2D Gaussians in a single forward pass, HyperGS combines the fast, flexible rendering of Gaussian Splatting with the speed and generalization of feedforward prediction, advancing Gaussians as a practical direction for fast and generalizable video representation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。