T5Gemma 2让轻量模型同时看图、读文、处理长文本。
T5Gemma 2: Seeing, Reading, and Understanding Longer
- 用UL2方法将单向解码器转为编码器-解码器架构，支持多模态输入。
- 在长文本任务上表现优于Gemma 3，270M模型仍保持高效。
- 适合需要多语言、多模态和长上下文理解的研究者使用。
我们提出T5Gemma 2，是T5Gemma系列新一代轻量级开源编码器-解码器模型，具备强大的多语言、多模态和长上下文能力。T5Gemma 2延续T5Gemma的适配策略（通过UL2），将预训练的仅解码器模型转化为编码器-解码器结构，并基于Gemma 3模型扩展至多模态。我们提出两种效率优化方法：共享编码器与解码器全部词嵌入的绑定词嵌入（tied word embedding），以及将解码器自注意力与交叉注意力合并为单一联合模块的融合注意力（merged attention）。实验表明该适配策略在不同架构与模态间具有通用性，且编码器-解码器结构在长上下文建模中表现突出。与T5Gemma类似，T5Gemma 2在预训练性能上相当或更优，后训练性能显著优于其Gemma 3对应版本。我们已发布270M-270M、1B-1B和4B-4B三个规模的预训练模型供社区研究使用。
原文摘要 · Abstract (English)
We introduce T5Gemma 2, the next generation of the T5Gemma family of lightweight open encoder-decoder models, featuring strong multilingual, multimodal and long-context capabilities. T5Gemma 2 follows the adaptation recipe (via UL2) in T5Gemma -- adapting a pretrained decoder-only model into an encoder-decoder model, and extends it from text-only regime to multimodal based on the Gemma 3 models. We further propose two methods to improve the efficiency: tied word embedding that shares all embeddings across encoder and decoder, and merged attention that unifies decoder self- and cross-attention into a single joint module. Experiments demonstrate the generality of the adaptation strategy over architectures and modalities as well as the unique strength of the encoder-decoder architecture on long context modeling. Similar to T5Gemma, T5Gemma 2 yields comparable or better pretraining performance and significantly improved post-training performance than its Gemma 3 counterpart. We release the pretrained models (270M-270M, 1B-1B and 4B-4B) to the community for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。