JEPA-T通过融合文本与图像令牌，实现高效图文生成。
JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation
- 用联合嵌入预测架构统一处理图文令牌，支持条件去噪。
- 在ImageNet-1K上实现强数据效率与开放词汇泛化能力。
- 适合关注图文对齐与生成通用性的研究者使用。
当前文本到图像（T2I）生成多依赖以标记为中心的自监督架构，但有效融合文本与视觉标记仍是挑战。我们提出 extbf{JEPA-T}，一种统一的多模态框架，将图像和标题编码为离散的视觉与文本标记，并由联合嵌入预测变换器处理。为增强融合，我们在特征预测器后引入交叉注意力以实现条件去噪，同时保持任务无关的主干网络。此外，原始文本嵌入在流匹配损失前注入，以提升训练过程中的对齐效果。推理时，同一网络可通过迭代去噪视觉标记，实现类别条件和自由文本图像生成。ImageNet-1K上的评估表明，JEPA-T具备强数据效率、开放词汇泛化能力，且持续优于非融合与晚融合基线。该方法显示，晚期架构融合结合目标级对齐，在条件强度与主干通用性之间提供了有效平衡。代码已开源：https://github.com/justin-herry/JEPA-T.git
原文摘要 · Abstract (English)
Modern Text-to-Image (T2I) generation increasingly relies on token-centric architectures that are trained with self-supervision, yet effectively fusing text with visual tokens remains a challenge. We propose \textbf{JEPA-T}, a unified multimodal framework that encodes images and captions into discrete visual and textual tokens, processed by a joint-embedding predictive Transformer. To enhance fusion, we incorporate cross-attention after the feature predictor for conditional denoising while maintaining a task-agnostic backbone. Additionally, raw texts embeddings are injected prior to the flow matching loss to improve alignment during training. During inference, the same network performs both class-conditional and free-text image generation by iteratively denoising visual tokens conditioned on text. Evaluations on ImageNet-1K demonstrate that JEPA-T achieves strong data efficiency, open-vocabulary generalization, and consistently outperforms non-fusion and late-fusion baselines. Our approach shows that late architectural fusion combined with objective-level alignment offers an effective balance between conditioning strength and backbone generality in token-based T2I.The code is now available: https://github.com/justin-herry/JEPA-T.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。