arXiv:2505.14671cs.CV2025-05NeurIPS被引 58

统一概念令牌提升个性化理解与生成能力

UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens

  • 用统一令牌融合理解与生成任务,共享语义信息
  • 在属性推理生成上达到当前最佳表现
  • 适合需要精细个性化图像生成的研究者

个性化模型在理解和生成用户提供的概念方面已取得显著成效。然而,现有方法对理解与生成使用独立的概念令牌,将两者孤立处理,可能导致复杂提示下的生成受限。例如,仅给定概念⟨bo⟩,就需生成“⟨bo⟩戴着它的帽子”而无需额外描述帽子。我们称此类生成为个性化属性推理生成。为此,本文提出UniCTokens框架,通过训练一组统一的概念令牌,将个性化信息有效融入统一视觉语言模型(VLM),以增强理解与生成能力。同时,设计三阶段渐进式训练策略:理解预热、基于理解的生成引导、基于生成的深度理解优化,实现双向促进。为量化评估统一VLM的个性化能力,我们构建了首个基准测试UnifyBench,涵盖概念理解、概念生成和属性推理生成。实验表明,UniCTokens在概念理解与生成上表现优异,并在个性化属性推理生成上达到领先水平。研究证明,理解增强生成,生成亦反哺理解。代码与数据集将公开于https://github.com/arctanxarc/UniCTokens。

原文摘要 · Abstract (English)

Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation \textit{\textbf{personalized attribute-reasoning generation}}. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{https://github.com/arctanxarc/UniCTokens}{https://github.com/arctanxarc/UniCTokens}.

个性化生成视觉语言模型统一令牌属性推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。