解决滚动快门视频导致的人体动画失真问题,重建清晰可动的3D avatar。
Scanline-Aware Animatable Gaussian Avatars from Rolling-Shutter Videos

- 用扫描线级时序建模替代传统帧级渲染,精准捕捉运动过程中的姿态变化。
- 在RS-ZJU数据集上,新方法使新视角合成效果媲美瞬时帧,提升显著。
- 适合需要高精度动态人体重建的虚拟人、影视特效和VR应用开发者。
可动画化的人体三维化身通常从多视角视频中重建,隐含假设是每帧中每个像素都对应同一时刻的身体动作。然而,滚动快门(RS)传感器逐行曝光,导致一帧内人物头部与脚部的动作相差数十毫秒,每条扫描线看到的都是不同姿态。将此类视频输入现有重建方法会把失真固化到标准表示中,表现为新视角和新姿态下的剪切与抖动。更严重的是,镜头阵列中各相机读出时间不一致,即使几何正确也破坏多视角一致性。本文提出RS-Avatar,直接从滚动快门视频重建清晰、无失真的可动画3D Gaussian avatar。核心思想仅需替换渲染操作:原本模糊模型对子帧渲染结果求平均,而滚动快门模型则按扫描线顺序拼接。这一微小改动即实现显著提升。在自建的RS-ZJU基准(基于ZJU-MoCap)上,该方法在所有受试者上均使新视角合成质量达到瞬时帧水平。
原文摘要 · Abstract (English)
Animatable human avatars are routinely reconstructed from multi-view video under a silent assumption: that every pixel of a frame observes the same instant of the body's motion. Rolling-shutter (RS) sensors expose image rows sequentially, so within one frame the head and the feet of a moving person are separated by tens of milliseconds of articulated motion, and every scanline sees a different pose. Feeding such video to a state-of-the-art avatar bakes the distortion into the canonical representation, where it survives as shear and wobble under novel views and novel poses. Worse, every camera in a rig follows its own readout schedule, so the multi-view consistency that drives the reconstruction is violated even when the geometry is correct. We present RS-Avatar, which reconstructs a sharp, undistorted, animatable 3D Gaussian avatar directly from RS video. The formulation is minimal: a motion-aware avatar already renders the body at several sub-frame instants, and where a blur model averages those renderings, a rolling-shutter model composites them scanline by scanline. Changing that operator is the only modification required. On RS-ZJU, a benchmark we build from ZJU-MoCap, this improves novel-view synthesis over training as if the frames were instantaneous, on every subject. A motion-aware blur model built on the same sub-frame machinery does not transfer, and in fact falls below the shutter-oblivious baseline: the machinery is reusable, the operator is not.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。