统一使用同一套面部骨骼框架,提升真人到虚拟角色的表演重定向精度。
Improving Facial Rig Semantics for Tracking and Retargeting
- 用统一骨架框架实现人物与虚拟角色间面部动作重定向。
- 通过西蒙说指令表情校准,使虚拟角色骨架匹配真实表演特征。
- 引入隐式微分微调追踪器,优化语义化动画控制,适合游戏/虚拟现实应用。
本文研究将捕捉到的面部表演重定向至另一人或游戏/虚拟现实(VR)中的虚拟角色。通过在表演者和目标上均采用相同的骨架框架(如3DMM、FLAME、MetaHuman等),避免了不同框架间语义对齐的难题。我们利用体素变形将面部骨架拟合到表演者和目标;同时,使用精心选择的Simon-Says表情序列校准每个骨架与其对应表演者或目标的运动特征。尽管通用的Simon-Says序列适用于人到人的重定向,但人到游戏/虚拟角色重定向时,采用反映角色特有运动特征的表情更优。经校准的骨架能正确生成预期表情,但在实际追踪过程中仍可能产生不理想的控制信号(良好函数的逆可能病态)。为此,我们提出一种基于隐式微分的微调方法,调整追踪器所用骨架,以生成更具语义意义的动画控制,从而提升重定向效果。该方法可处理非可微追踪器,适用于真实场景。
原文摘要 · Abstract (English)
In this paper, we consider retargeting a tracked facial performance to either another person or to a virtual character in a game or virtual reality (VR) environment. We remove the difficulties associated with identifying and retargeting the semantics of one rig framework to another by utilizing the same framework (3DMM, FLAME, MetaHuman, etc.) for both subjects. Although this does not constrain the choice of framework when retargeting from one person to another, it does force the tracker to use the game/VR character rig when retargeting to a game/VR character. We utilize volumetric morphing in order to fit facial rigs to both performers and targets; in addition, a carefully chosen set of Simon-Says expressions is used to calibrate each rig to the motion signatures of the relevant performer or target. Although a uniform set of Simon-Says expressions can likely be used for all person to person retargeting, we argue that person to game/VR character retargeting benefits from Simon-Says expressions that capture the distinct motion signature of the game/VR character rig. The Simon-Says calibrated rigs tend to produce the desired expressions when exercising animation controls (as expected). Unfortunately, these well-calibrated rigs still lead to undesirable controls when tracking a performance (a well-behaved function can have an arbitrarily ill-conditioned inverse), even though they typically produce acceptable geometry reconstructions. Thus, we propose a fine-tuning approach that modifies the rig used by the tracker in order to promote the output of more semantically meaningful animation controls, facilitating high efficacy retargeting. In order to better address real-world scenarios, the fine-tuning relies on implicit differentiation so that the tracker can be treated as a (potentially non-differentiable) black box.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。