无需合成目标图,仅用1万张参考图即可实现人物形象个性化生成。
CRAFT: Constrained Reward via Attention Fine-Tuning for Subject Personalization without Composed Targets

- 通过注意力机制约束奖励,让模型聚焦正确参考主体。
- 仅用1万张参考图像和掩码,达到顶尖性能。
- 方法通用,可迁移至多种模型,适合内容创作场景。
主体驱动的图像个性化——在新场景中生成保留参考主体身份的新图像——是现代视觉内容创作的核心能力。当前主流方法依赖大规模成对数据(参考图像与合成目标图像),需经过多阶段繁琐流程生成合成目标,且高度耦合于特定合成器。本文提出CRAFT(基于注意力微调的约束奖励),采用单步ReFL框架,仅使用10,000张参考图像与主体掩码,无需任何合成目标监督,通过LoRA适配器微调预训练的参考感知多模态扩散变换器(MMDiT)。CRAFT实现“关注何处”原则:在注意力层级上对齐噪声与词元注意力,使每个主体的注意力掩码控制像素级身份奖励,确保图像空间监督与学习到的注意力路由一致。应用于FLUX.2-klein-9B,在XVerseBench上达到当前最优表现,而此前方法需15万至超过200万合成对。该方案可迁移至其他参考感知模型,持续提升性能。
原文摘要 · Abstract (English)
Subject-driven image personalization---generating new images that preserve the identity of one or several reference subjects in novel scenes---is a foundational capability for modern visual content creation. It is currently dominated by generalized methods that fine-tune a pretrained multimodal diffusion transformer (MMDiT) on hundreds of thousands to millions of paired \emph{(reference, composed-target)} examples, where each composed target is a synthesized image of the subject in a novel scene. Producing such targets demands a costly multi-stage curation pipeline---LLM-based prompt generation, T2I-based composed-target synthesis, reference-subject extraction, VLM-based quality filtering, and correspondence labeling---and tightly couples each method to a particular target synthesizer and curation choice. We introduce \emph{CRAFT} (Constrained Reward via Attention Fine-Tuning), a single-step ReFL framework that fine-tunes a pre-trained \emph{reference-aware} MMDiT via LoRA adapters using a compact reference-only data construction---$10$K reference images and subject masks, with no composed-target supervision. CRAFT realizes a \emph{Where to look} principle: attention-level rewards align noise- and phrase-token attention with the correct reference subject, and the resulting per-subject attention masks gate a pixel-level identity reward to keep image-space supervision consistent with the learned attention routing. Applied to FLUX.2-klein-9B, CRAFT achieves state-of-the-art performance on XVerseBench \rev{while using no composed-target supervision---only $10$K reference-only samples, whereas prior generalized methods require $150$K to over $2$M composed-target pairs}. The same recipe transfers to other reference-aware backbones, consistently improving performance. Project page: https://jihun999.github.io/projects/CRAFT/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。