提出高效评估文本编码器的框架,提升文生图模型语义准确性。
GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models
- 构建无需训练的文本基准测试集,750倍提速效果评估。
- 两阶段训练法优化文本特征,生成更精准的图像与视频。
- 适合关注文生图质量提升的研究者与开发者。
文本编码器是文生图与文生视频扩散模型的关键组件,直接影响生成内容的语义保真度。然而其发展受限于两大挑战:缺乏能可靠预测下游生成性能的高效评估框架,以及难以有效适配预训练语言模型用于视觉合成。为此,我们提出GRAN-TED——一种为扩散模型生成鲁棒、对齐且细腻文本嵌入的新范式。贡献有二:首先,提出TED-6K这一纯文本基准测试集,可在不进行端到端训练的情况下高效稳健地评估编码器表征质量;通过轻量统一适配器标准化后,该基准与下游生成任务表现强相关。实验表明,在相同设置下,使用TED-6K评估比从头训练扩散模型快约750倍。其次,基于此验证框架,我们采用新型两阶段训练策略开发出更优文本编码器:先在多模态大模型上微调以获得更好视觉表征,再通过逐层加权提取更精细强大的文本特征。实验显示,所提GRAN-TED编码器不仅在TED-6K上达当前最优,还在文生图与文生视频任务中带来显著性能提升。TED-6K数据集与评估代码已公开:https://anonymous.4open.science/r/GRAN-TED-4FCC/。
原文摘要 · Abstract (English)
The text encoder is a critical component of text-to-image and text-to-video diffusion models, fundamentally determining the semantic fidelity of the generated content. However, its development has been hindered by two major challenges: the lack of an efficient evaluation framework that reliably predicts downstream generation performance, and the difficulty of effectively adapting pretrained language models for visual synthesis. To address these issues, we introduce GRAN-TED, a paradigm to Generate Robust, Aligned, and Nuanced Text Embeddings for Diffusion models. Our contribution is twofold. First, we propose TED-6K, a novel text-only benchmark that enables efficient and robust assessment of an encoder's representational quality without requiring costly end-to-end model training. We demonstrate that performance on TED-6K, standardized via a lightweight, unified adapter, strongly correlates with an encoder's effectiveness in downstream generation tasks. Notably, under our experimental setup, compared with training a diffusion model from scratch, evaluating with TED-6K is about \textbf{750$\times$ faster}. Second, guided by this validated framework, we develop a superior text encoder using a novel two-stage training paradigm. This process involves an initial fine-tuning stage on a Multimodal Large Language Model for better visual representation, followed by a layer-wise weighting method to extract more nuanced and potent text features. Our experiments show that the resulting GRAN-TED encoder not only achieves state-of-the-art performance on TED-6K but also leads to demonstrable performance gains in text-to-image and text-to-video generation. Our TED-6K dataset and evaluation code are available at the following link: https://anonymous.4open.science/r/GRAN-TED-4FCC/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。