用文字控制视觉嵌入,让模型可自由调整颜色、风格等属性。
Controlling Embedding Spaces with Text-Conditioned Transformations

- 通过文本条件生成仿射变换,显式操控嵌入空间中的属性。
- 在不重新编码的前提下实现属性检索,推理成本几乎为零。
- 适用于多属性组织与解耦,兼容现有大规模嵌入模型。
CLIP等多模态嵌入模型虽能实现语义相似性检索与跨模态零样本分类,但其嵌入向量常仅表达主导语义(如主体对象),忽略相机角度、色调等重要属性。本文提出一种文本条件化的视觉嵌入变换方法,输入如“颜色”或“艺术风格”等属性描述,网络生成相应仿射变换以突出指定属性。该机制通过自然语言条件同时学习多种属性,可在推理时通过直观接口灵活调用。训练目标为对齐变换后嵌入与冻结的潜在空间,实现无需重编码即可使用现有大规模嵌入进行检索。当应用于全集时,同一机制可用于多聚类等属性解耦任务。由于直接在潜在空间操作,该方法提供统一高效的嵌入空间控制框架,在属性检索与多属性组织任务中均达到当前最优性能,且推理开销近乎为零。
原文摘要 · Abstract (English)
Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level semantics into a single vector, which comes at the cost of primarily expressing a dominant semantics like main object while suppressing other important attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., "color" or "art style"), a network generates an affine transformation that emphasizes the specified attribute. Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost. Project page: https://joefioresi718.github.io/ControlEmbed_webpage/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。