通过对齐文本标记语义,实现图像生成中平滑的语义过渡。
Token-to-Token Alignment of Text Embeddings for Semantic Blending

- 构建标记到标记的语义对应关系,统一不同提示词的结构
- 对齐后线性插值可产生连贯的语义变化,支持图像融合与连续编辑
- 无需修改模型即可实现精准语义控制,适合内容创作与编辑场景
在现代生成模型中,图像由文本提示词的标记序列控制。然而,标记序列空间缺乏一致的可访问结构:语义相似的图像可能对应词语、顺序或概念位置不同的序列,而相似的标记序列也可能表达完全不同语义。这种结构缺失使得该空间中的平滑过渡难以实现,阻碍了图像融合和连续编辑等应用。我们认为问题不在于语义结构本身缺失,而在于表示之间的错位。为此,我们提出标记到标记对齐框架,建立跨提示词间标记的显式语义对应。方法分为两阶段:首先将提示词重构为共享结构形式(结构对齐),再基于语义相似性对齐标记嵌入(嵌入级对齐)。对齐后,简单的线性插值成为有意义操作,生成平滑连贯的语义过渡,支持图像融合与连续编辑。结果表明,文本到图像模型中的文本嵌入空间隐含连续语义结构,一旦表示被正确对齐,该结构即被激活。这意味着通过组织现有表示即可实现语义控制,而无需修改生成模型。
原文摘要 · Abstract (English)
In modern generative models, images are specified and controlled through text prompts. In practice, images are generated from sequences of tokens derived from these prompts. However, the space of token sequences lacks a consistent accessible structure: semantically similar images may correspond to sequences that differ in wording, ordering, and placement of concepts, while similar token sequences may encode very different semantics. This apparent lack of structure makes it difficult to perform smooth transitions in this space, hindering applications such as image blending and continuous control of edits. We argue that this limitation stems not from the absence of semantic structure, but from misalignment between representations. To address this misalignment, we introduce Token-to-Token alignment, a framework that establishes explicit semantic correspondence between tokens across prompts. Our approach transforms prompts into a structured representation in which semantically corresponding concepts are mapped to consistent positions across prompts, and then aligns their token embeddings based on semantic similarity. Concretely, the method consists of two stages: a structural alignment that rephrases prompts into a shared structured form, followed by an embedding-level alignment that matches token representations across prompts. With this alignment in place, simple linear interpolation becomes a meaningful operation, producing smooth and coherent semantic transitions and enabling applications such as blending and continuous editing. Our results show that text embedding spaces in text-to-image models implicitly encode a continuous semantic structure that becomes accessible once representations are properly aligned, suggesting that semantic control can be achieved by organizing existing representations rather than modifying the generative model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。