轻量级模型实现高动态光照图的精细纹理重建。
VQT-Light:Lightweight HDR Illumination Map Prediction with Richer Texture.pdf
- 用VQVAE提取离散特征防后验崩溃，用ViT捕获全局上下文。
- 推理速度达40FPS，多指标优于现有方法。
- 适合实时渲染与移动设备上的光照估计应用。
准确的光照估计在计算机视觉与图形学中至关重要但极具挑战。现有方法或难以恢复光照图的细节纹理，或存在运行速度慢、纹理保真度低的问题。为此，我们提出基于VQVAE与ViT架构的轻量级框架VQT-Light，包含特征提取与光照估计两个模块。首先，利用VQVAE提取光照图的离散特征，避免“后验崩溃”；其次，采用ViT而非CNN捕捉输入图像的全局上下文与依赖关系，提升视场外光照预测能力。将光照估计建模为多分类任务，显著提升性能。实验表明，VQT-Light在保持40FPS推理速度的同时，大幅提升了多项评估指标，定性与定量结果均优于现有先进方法。
原文摘要 · Abstract (English)
Accurate lighting estimation is a significant yet challenging task in computer vision and graphics. However, existing methods either struggle to restore detailed textures of illumination map, or face challenges in running speed and texture fidelity. To tackle this problem, we propose a novel framework (VQT-Light) based on VQVAE and ViT architecture. VQT-Light includes two modules: feature extraction and lighting estimation. First, we take advantages of VQVAE to extract discrete features of illumination map rather than continuous features to avoid "posterior collapse". Second, we capture global context and dependencies of input image through ViT rather than CNNs to improve the prediction of illumination outside the field of view. Combining the above two modules, we formulate the lighting estimation as a multiclass classification task, which plays a key role in our pipeline. As a result, our model predicts light map with richer texture and better fidelity while keeping lightweight and fast. VQT-Light achieves an inference speed of 40FPS and improves multiple evaluation metrics. Qualitative and quantitative experiments demonstrate that the proposed method realizes superior results compared to existing state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。