用单个[IMG]标记高效生成图文设计，解决多图融合瓶颈
Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design
- 用双浅层MLP块+压缩-扩展结构，将文本隐状态映射到视觉嵌入空间
- 仅用一个[IMG] token表示每张图，输入长度大幅降低，支持千级元素融合
- 轻量设计适合复杂图形生成，尤其擅长图文混排与布局协同
多模态大语言模型在理解多媒体内容上取得进展，但生成能力仍受限。现有方法通过将文本标记的隐藏表示转换为视觉模型嵌入或原始图像数据来弥补差距，但通常需多个专用标记表示每张图，显著增加输入长度。这在图形设计生成等需融合数千个文本、图像和布局信息的任务中成为主要瓶颈。为此，提出一种新架构，使用单个[IMG]标记将文本隐藏表示映射至视觉模型（如CLIP ViT-L/14）嵌入空间。该架构采用两个浅层MLP块，各自含独立压缩模块及共享扩展模块，训练时结合六种损失函数，一模块辅助另一模块训练，推理时移除，实现轻量化。在图像到设计和文本到设计生成任务中均表现强劲。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting mul- timedia content. However, their ability to generate me- dia remains limited. Recent approaches have attempted to bridge this gap by translating the hidden representations of token sequences into the embedding space of visual models or directly into raw image data. However, these methods often represent each image using multiple specialised to- kens which significantly increases the input length. This be- comes a major limitation for tasks such as graphic design generation where the output typically involves a seamless blend of thousands of tokens across text, multiple images, and layout information. To address this challenge, a novel architecture is proposed that maps hidden token represen- tations to the embedding space of visual models, such as CLIP ViT-L/14, using a single [IMG] token per image. The architecture employs two shallow MLP blocks, each with a separate compression module followed by a shared expan- sion module, trained with six distinct loss functions. One block aids the other during training and is omitted during inference, resulting in a lightweight solution. Strong perfor- mance is demonstrated in both image-to-design and text-to- design generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。