解决生成个性化中的语义坍缩问题，提升图像与提示词的语义一致性。
Mitigating Semantic Collapse in Generative Personalization with Test-Time Embedding Adjustment
- 推理时动态调整预训练嵌入的大小和方向，防止语义漂移。
- 在多概念提示下显著改善文本-图像对齐，避免输出简化为单一概念。
- 无需训练、兼容多种个性化方法，适合希望提升生成质量的研究者。
本文研究生成个性化中的语义坍缩问题，即学习到的视觉概念 $V$ 逐渐偏离原始文本语义，在多概念提示中主导其他概念。这导致复杂提示如“一张 $V$ 戴眼镜弹吉他的照片”被简化为“一张 $V$ 的照片”，输出图像也失去上下文丰富性。我们发现根源在于无约束优化使嵌入 $V$ 在嵌入空间中任意漂移（方向与幅度）。为此，提出一种无需训练的推理时嵌入调整方法，动态修正 $V$ 的方向与大小，有效缓解语义坍缩。该方法广泛适用于多种个性化方法，在多样场景中显著提升文本-图像对齐效果。代码已匿名发布于 https://github.com/tuananhbui89/Embedding-Adjustment。
原文摘要 · Abstract (English)
In this paper, we investigate the semantic collapsing problem in generative personalization, an under-explored topic where the learned visual concept ($V$) gradually shifts from its original textual meaning and comes to dominate other concepts in multi-concept input prompts. This issue not only reduces the semantic richness of complex input prompts like "a photo of $V$ wearing glasses and playing guitar" into simpler, less contextually rich forms such as "a photo of $V$" but also leads to simplified output images that fail to capture the intended concept. We identify the root cause as unconstrained optimisation, which allows the learned embedding $V$ to drift arbitrarily in the embedding space, both in direction and magnitude. To address this, we propose a simple yet effective training-free method that adjusts the magnitude and direction of pre-trained embedding at inference time, effectively mitigating the semantic collapsing problem. Our method is broadly applicable across different personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. Our code is anonymously published at https://github.com/tuananhbui89/Embedding-Adjustment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。