arXiv:2503.09556cs.CV2025-03CVPR被引 3

用肌电图重建面部表情,解决遮挡难题。

Electromyography-Informed Facial Expression Reconstruction for Physiological-Based Synthesis and Analysis

  • 结合3D人脸模型与无配对图像转换,分离几何与外观特征。
  • 实现肌电活动与面部表情的双向映射,重建效果精准。
  • 适合心理、医疗及影视领域中多模态面部数据研究者。

肌肉活动与面部表情的关系在心理学、医学和娱乐等领域至关重要。通过表面肌电图(sEMG)同步记录面部模仿与肌肉活动,可深入理解这些复杂动态。然而,现有方法无法处理电极遮挡问题,即使有同一人的无遮挡参考图像,表情强度与执行差异仍难以匹配。本文提出肌电图引导的面部表情重建(EIFER)方法,采用对抗方式在sEMG遮挡下忠实还原面部。通过将3D可变形模型(3DMM)与基于参考记录的神经无配对图像到图像转换相结合,解耦面部几何与视觉外观(如皮肤纹理、光照、电极)。EIFER学习3DMM表达参数与肌肉活动之间的双向映射,建立两域对应关系。实验基于同步的sEMG与面部模仿数据集验证了该方法的有效性,实现了高保真几何与外观重建。此外,还可基于肌肉活动合成表情,并从观察表情反推动态肌电活动。EIFER为面部肌电图开辟新范式,可扩展至其他多模态面部记录场景。

原文摘要 · Abstract (English)

The relationship between muscle activity and resulting facial expressions is crucial for various fields, including psychology, medicine, and entertainment. The synchronous recording of facial mimicry and muscular activity via surface electromyography (sEMG) provides a unique window into these complex dynamics. Unfortunately, existing methods for facial analysis cannot handle electrode occlusion, rendering them ineffective. Even with occlusion-free reference images of the same person, variations in expression intensity and execution are unmatchable. Our electromyography-informed facial expression reconstruction (EIFER) approach is a novel method to restore faces under sEMG occlusion faithfully in an adversarial manner. We decouple facial geometry and visual appearance (e.g., skin texture, lighting, electrodes) by combining a 3D Morphable Model (3DMM) with neural unpaired image-to-image translation via reference recordings. Then, EIFER learns a bidirectional mapping between 3DMM expression parameters and muscle activity, establishing correspondence between the two domains. We validate the effectiveness of our approach through experiments on a dataset of synchronized sEMG recordings and facial mimicry, demonstrating faithful geometry and appearance reconstruction. Further, we synthesize expressions based on muscle activity and how observed expressions can predict dynamic muscle activity. Consequently, EIFER introduces a new paradigm for facial electromyography, which could be extended to other forms of multi-modal face recordings.

肌电图表情重建多模态3DMM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。