arXiv:2512.13672cs.LGcs.CV2025-12被引 2

解决文本生成个性化中提示失真的问题,提升图像与提示的匹配度。

Directional Textual Inversion for Personalized Text-to-Image Generation

  • 只优化嵌入方向,固定大小,避免数值膨胀
  • 在多个任务上提升文本保真度,保持主体一致性
  • 支持语义连贯插值,适合创意生成场景

文本反转(TI)是一种高效的文本到图像个性化方法,但在复杂提示下表现不佳。我们发现失败源于嵌入范数膨胀:学习到的标记漂移到分布外幅度,削弱了预归一化变换器中的提示条件。实证表明,语义主要由CLIP标记空间中的方向编码,而膨胀的范数损害上下文表达;理论分析显示大范数会衰减位置信息并阻碍残差更新。我们提出方向性文本反转(DTI),将嵌入幅度固定在分布内尺度,仅在单位超球面上通过黎曼SGD优化方向。将方向学习建模为带冯·米塞斯-费舍尔先验的MAP,得到简单高效的恒定方向先验梯度。在各类个性化任务中,DTI在保持主体相似性的同时,优于TI及其变体的文本保真度。关键优势在于超球面参数化支持平滑、语义连贯的概念插值(slerp),这是标准TI所不具备的能力。研究表明,仅优化方向是实现提示忠实个性化的稳健且可扩展路径。代码已开源。

原文摘要 · Abstract (English)

Textual Inversion (TI) is an efficient approach to text-to-image personalization but often fails on complex prompts. We trace these failures to embedding norm inflation: learned tokens drift to out-of-distribution magnitudes, degrading prompt conditioning in pre-norm Transformers. Empirically, we show semantics are primarily encoded by direction in CLIP token space, while inflated norms harm contextualization; theoretically, we analyze how large magnitudes attenuate positional information and hinder residual updates in pre-norm blocks. We propose Directional Textual Inversion (DTI), which fixes the embedding magnitude to an in-distribution scale and optimizes only direction on the unit hypersphere via Riemannian SGD. We cast direction learning as MAP with a von Mises-Fisher prior, yielding a constant-direction prior gradient that is simple and efficient to incorporate. Across personalization tasks, DTI improves text fidelity over TI and TI-variants while maintaining subject similarity. Crucially, DTI's hyperspherical parameterization enables smooth, semantically coherent interpolation between learned concepts (slerp), a capability that is absent in standard TI. Our findings suggest that direction-only optimization is a robust and scalable path for prompt-faithful personalization. Code is available at https://github.com/kunheek/dti.

文本生成个性化方向优化扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。