不用训练，仅用CLIP就能生成图像，还能做风格迁移和重建。
Implicit Inversion turns CLIP into a Decoder
- 用隐式神经表示逆向映射CLIP的共享空间，分层控制频率生成。
- 无需训练或微调，直接实现文本到图像生成、风格迁移和图像重建。
- 适合对生成模型原理感兴趣的研究者，或想快速搭建生成系统的人。
CLIP 是一种判别式模型，通过在共享嵌入空间中对齐图像与文本而训练。由于其多模态结构，常作为生成流水线的骨干，需额外训练解码器将共享空间映射回图像。本文证明，仅使用 CLIP 本身即可实现图像合成——无需解码器、训练或微调。我们的方法优化了一个频域感知的隐式神经表示，通过分层安排网络各层的频率，实现从粗到细的生成过程。为稳定逆映射，引入对抗鲁棒初始化、轻量级正交普罗克鲁斯特斯投影以对齐局部图文嵌入，并采用锚定输出至自然图像统计特性的混合损失。在不修改 CLIP 权重的前提下，该框架实现了文本到图像生成、风格迁移和图像重建等能力。研究提示，判别模型可能蕴含未被发掘的生成潜力，就在眼前却鲜为人知。
原文摘要 · Abstract (English)
CLIP is a discriminative model trained to align images and text in a shared embedding space. Due to its multimodal structure, it serves as the backbone of many generative pipelines, where a decoder is trained to map from the shared space back to images. In this work, we show that image synthesis is nevertheless possible using CLIP alone -- without any decoder, training, or fine-tuning. Our approach optimizes a frequency-aware implicit neural representation that encourages coarse-to-fine generation by stratifying frequencies across network layers. To stabilize this inverse mapping, we introduce adversarially robust initialization, a lightweight Orthogonal Procrustes projection to align local text and image embeddings, and a blending loss that anchors outputs to natural image statistics. Without altering CLIP's weights, this framework unlocks capabilities such as text-to-image generation, style transfer, and image reconstruction. These findings suggest that discriminative models may hold untapped generative potential, hidden in plain sight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。