arXiv:2511.02830cs.CV2025-11

用点轨迹学习人脸像素的3D规范嵌入,实现高精度对应匹配。

Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks

  • 基于视觉变换器预测每像素在3D单位立方体中的嵌入坐标。
  • 在真实说话人视频上训练,实现姿态变化下全头(含头发)一致匹配。
  • 适用于人脸追踪、立体重建,适合需要语义对齐的研究者。

我们提出DenseMarks——一种面向人脸图像的新式可学习表示,支持高质量的密集对应关系。对于一张人脸2D图像,视觉变换器网络为每个像素预测一个3D嵌入,对应于3D规范单位立方体中的位置。为训练该网络,我们在多样化的野生说话人视频上收集成对点匹配数据,通过状态领先点追踪器估计,并以对比损失引导匹配点具有相近嵌入。进一步结合多任务学习,引入人脸关键点与分割约束,并通过潜在立方体特征保证嵌入的空间连续性,形成可解释且可查询的规范空间。该表示可用于识别共同语义部位、人脸/头部追踪及立体重建。得益于强监督,方法对姿态变化鲁棒,覆盖整个头部(包括头发)。规范空间瓶颈确保了跨不同姿态和个体的一致性表示。在几何感知点匹配和单目头模型追踪任务中达到当前最优性能。代码与模型检查点将公开发布。

原文摘要 · Abstract (English)

We propose DenseMarks - a new learned representation for human heads, enabling high-quality dense correspondences of human head images. For a 2D image of a human head, a Vision Transformer network predicts a 3D embedding for each pixel, which corresponds to a location in a 3D canonical unit cube. In order to train our network, we collect a dataset of pairwise point matches, estimated by a state-of-the-art point tracker over a collection of diverse in-the-wild talking heads videos, and guide the mapping via a contrastive loss, encouraging matched points to have close embeddings. We further employ multi-task learning with face landmarks and segmentation constraints, as well as imposing spatial continuity of embeddings through latent cube features, which results in an interpretable and queryable canonical space. The representation can be used for finding common semantic parts, face/head tracking, and stereo reconstruction. Due to the strong supervision, our method is robust to pose variations and covers the entire head, including hair. Additionally, the canonical space bottleneck makes sure the obtained representations are consistent across diverse poses and individuals. We demonstrate state-of-the-art results in geometry-aware point matching and monocular head tracking with 3D Morphable Models. The code and the model checkpoint will be made available to the public.

人脸建模嵌入学习密集对应3D表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。