arXiv:2505.02108cs.CV2025-05被引 5

用高斯点云渲染手语,精准捕捉细微手部与面部动作。

SignSplat: Rendering Sign Language via Gaussian Splatting

  • 基于稀疏多视角数据,通过约束网格参数提升手语动作建模精度。
  • 在复杂手语序列上实现优于现有方法的视觉保真度与一致性。
  • 适合需要高精度人体细节建模的虚拟人、手语合成场景。

当前基于高斯点云的条件人体渲染方法主要关注舞蹈或行走等简单动作,但手语等复杂场景更关注手部与面部的细微运动。由于多视角手语数据采集困难,现有方法受限于视角数量。本文提出利用时序数据中的动态信息,通过约束网格参数构建高保真渲染框架,结合高斯参数正则化抑制过拟合与渲染伪影,并设计自适应密度控制策略以动态优化点云分布。实验表明,在基准数据集上达到领先性能;在高度精细的手语动作序列中,显著优于现有方法,支持高质量手语视频新视角生成,融合神经机器翻译实现手语片段拼接。

原文摘要 · Abstract (English)

State-of-the-art approaches for conditional human body rendering via Gaussian splatting typically focus on simple body motions captured from many views. This is often in the context of dancing or walking. However, for more complex use cases, such as sign language, we care less about large body motion and more about subtle and complex motions of the hands and face. The problems of building high fidelity models are compounded by the complexity of capturing multi-view data of sign. The solution is to make better use of sequence data, ensuring that we can overcome the limited information from only a few views by exploiting temporal variability. Nevertheless, learning from sequence-level data requires extremely accurate and consistent model fitting to ensure that appearance is consistent across complex motions. We focus on how to achieve this, constraining mesh parameters to build an accurate Gaussian splatting framework from few views capable of modelling subtle human motion. We leverage regularization techniques on the Gaussian parameters to mitigate overfitting and rendering artifacts. Additionally, we propose a new adaptive control method to densify Gaussians and prune splat points on the mesh surface. To demonstrate the accuracy of our approach, we render novel sequences of sign language video, building on neural machine translation approaches to sign stitching. On benchmark datasets, our approach achieves state-of-the-art performance; and on highly articulated and complex sign language motion, we significantly outperform competing approaches.

手语生成高斯溅射人体渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。