arXiv:2605.12271cs.CV2026-05

用户用图像页直接生成图像,跳过文字描述瓶颈。

Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm

论文配图:Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm
图 1 · 摘自论文原文
  • 用视觉页面替代文本作为生成条件,直接传递视觉意图。
  • V2V-Zero在GenEval上达0.85分,接近优化后文本生成效果。
  • 适合需要精准视觉控制的设计师、艺术家及跨模态研究者。

人类常通过字体稿、草图、参考图和标注场景等视觉素材表达与创作意图,但当前视觉生成模型仍要求用户将意图转化为文本,压缩了空间结构、外观细节和字形特征等关键信息。本文提出视觉到视觉(V2V)生成范式,用户以视觉文档为条件生成目标内容。提出V2V-Zero框架,无需训练即可在现有视觉语言模型(VLM)基础上实现该接口:将文本条件替换为从视觉页面提取的最终层隐藏状态,利用冻结的VLM已有的图文映射能力。在GenEval评测中,使用冻结的Qwen-Image骨干网络,得分达0.85,接近其优化后的文本生成表现。为进一步评估V2V潜力,构建Simple-V2V Bench,涵盖7类视觉条件任务与7种模型(含GPT Image 2、Nano Banana 2、Seedream 5.0 Lite、开源基线及视频扩展)。V2V-Zero得分为32.7/100,优于所测开源图像基线,揭示能力层级:属性绑定强,内容生成不稳定,结构控制仍难,即便商业系统亦然。基于HunyuanVideo-1.5的视频扩展得分为20.2/100,证明该接口可泛化至视频生成。机制分析显示,默认推理路径主要依赖视觉信号,95.0%的条件标记注意力集中于视觉页面隐藏状态。

原文摘要 · Abstract (English)

Humans often specify and create through visual artifacts: typography sheets, sketches, reference images, and annotated scenes. Yet modern visual generators still ask users to serialize this intent into text, a bottleneck that compresses signals like spatial structure, exact appearance, and glyph shape. We propose \textbf{\emph{visual-to-visual} (V2V)} generation, in which the user conditions a generative model with a visual specification page rather than a text prompt. The page is not an edit target, but a visual document that specifies the desired output. We introduce \textbf{V2V-Zero}, a training-free framework that exposes this interface in existing vision-language model (VLM) conditioned generators by replacing text-only conditioning with final-layer hidden states extracted from visual pages, exploiting the fact that the frozen VLM already maps both text and images into the generator's conditioning space. On GenEval, V2V-Zero reaches 0.85 with a frozen Qwen-Image backbone, closely matching its optimized text-to-image performance without fine-tuning. To evaluate the broader V2V space, we introduce \textbf{Simple-V2V Bench}, spanning seven visual-conditioning tasks and seven models, including GPT Image 2, Nano Banana 2, Seedream 5.0 Lite, open-weight baselines, and a video extension. V2V-Zero scores 32.7/100, outperforming evaluated open-weight image baselines and revealing a clear capability hierarchy: attribute binding is strong, content generation is unreliable, and structural control remains hard even for commercial systems. A HunyuanVideo-1.5 extension scores 20.2/100, showing the interface transfers beyond images. Mechanistic analysis shows the default reasoning path is primarily visually routed, with 95.0\% of conditioning-token attention mass on visual-page hidden states.

视觉生成多模态生成模型无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。