提出TC-Light模型，实现高效且时序一致的视频重渲染。
TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer
- 分两阶段优化：全局光照对齐与精细纹理/光照对齐
- 在长时动态视频上实现物理合理重渲染，计算开销低
- 适用于模拟到真实、真实到真实的数据扩展场景
光照与纹理编辑是世界间迁移的关键，对模拟到真实及真实数据扩增等具身智能应用至关重要。现有生成式重渲染方法受限于训练数据域（如人像）或面临时序一致性与计算效率瓶颈，尤其在复杂动态和长时视频中表现不佳。本文提出TC-Light，一种新型生成式渲染器：首先通过扩展的视频重光照模型初步重光照，第一阶段优化外观嵌入以对齐全局光照；第二阶段优化提出的规范视频表示——唯一视频张量（UVT），以对齐细粒度纹理与光照。为全面评估性能，我们构建了长且高度动态的视频基准测试。大量实验表明，该方法可实现物理合理的重渲染结果，具备优异时序一致性与低计算成本。代码与视频演示见 https://dekuliutesla.github.io/tclight/。
原文摘要 · Abstract (English)
Illumination and texture editing are critical dimensions for world-to-world transfer, which is valuable for applications including sim2real and real2real visual data scaling up for embodied AI. Existing techniques generatively re-render the input video to realize the transfer, such as video relighting models and conditioned world generation models. Nevertheless, these models are predominantly limited to the domain of training data (e.g., portrait) or fall into the bottleneck of temporal consistency and computation efficiency, especially when the input video involves complex dynamics and long durations. In this paper, we propose TC-Light, a novel generative renderer to overcome these problems. Starting from the video preliminarily relighted by an inflated video relighting model, it optimizes appearance embedding in the first stage to align global illumination. Then it optimizes the proposed canonical video representation, i.e., Unique Video Tensor (UVT), to align fine-grained texture and lighting in the second stage. To comprehensively evaluate performance, we also establish a long and highly dynamic video benchmark. Extensive experiments show that our method enables physically plausible re-rendering results with superior temporal coherence and low computation cost. The code and video demos are available at https://dekuliutesla.github.io/tclight/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。