用关键点分词实现精细人脸编辑,保持身份不变且控制更准。
LaTo: Landmark-tokenized Diffusion Transformer for Fine-grained Human Face Editing
- 将人脸关键点直接转为离散标记,无需像素级对应。
- 身份保留提升7.8%,语义一致性提升4.6%。
- 适合需要精确表情/姿态调整的图像编辑场景。
基于指令的人脸编辑模型虽能实现语义操控,但在属性精确控制和身份保留方面仍存在挑战。现有方法常将关键点视为刚性几何约束,当目标关键点与源图差异较大(如大幅表情或姿态变化、关键点估计不准)时,易导致身份失真。为此,我们提出LaTo:一种用于细粒度、身份保真的人脸编辑的地标分词扩散变换器。核心创新包括:(1) 地标分词器,可直接将原始关键点坐标量化为离散面部标记,避免密集像素对应;(2) 位置映射位置编码与地标感知无分类器引导,联合实现指令、几何与外观的灵活解耦交互,强化身份保留;(3) 基于视觉-语言模型的地标预测器,从指令与源图推断目标关键点,其结构化思维链提升了估计精度与交互可控性。为缓解数据稀缺问题,我们构建了目前最大的该任务基准数据集HFL-150K,包含超过15万张真实人脸对及细粒度指令。大量实验表明,LaTo在身份保留上优于当前最优方法7.8%,在语义一致性上提升4.6%。代码与数据集将在论文接受后公开。
原文摘要 · Abstract (English)
Recent multimodal models for instruction-based face editing enable semantic manipulation but still struggle with precise attribute control and identity preservation. Structural facial representations such as landmarks are effective for intermediate supervision, yet most existing methods treat them as rigid geometric constraints, which can degrade identity when conditional landmarks deviate significantly from the source (e.g., large expression or pose changes, inaccurate landmark estimates). To address these limitations, we propose LaTo, a landmark-tokenized diffusion transformer for fine-grained, identity-preserving face editing. Our key innovations include: (1) a landmark tokenizer that directly quantizes raw landmark coordinates into discrete facial tokens, obviating the need for dense pixel-wise correspondence; (2) a location-mapped positional encoding and a landmark-aware classifier-free guidance that jointly facilitate flexible yet decoupled interactions among instruction, geometry, and appearance, enabling strong identity preservation; and (3) a landmark predictor that leverages vision-language models to infer target landmarks from instructions and source images, whose structured chain-of-thought improves estimation accuracy and interactive control. To mitigate data scarcity, we curate HFL-150K, to our knowledge the largest benchmark for this task, containing over 150K real face pairs with fine-grained instructions. Extensive experiments show that LaTo outperforms state-of-the-art methods by 7.8% in identity preservation and 4.6% in semantic consistency. Code and dataset will be made publicly available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。