Gemma 3扩展为多模态模型，支持长文本与多语言，性能媲美更大模型。
Gemma 3 Technical Report
- 采用局部注意力占比提升的架构，降低长文本下的显存占用。
- 40亿参数版本在指令微调后性能接近270亿参数的Gemma2。
- 支持128K以上上下文，数学、对话等能力显著提升，适合本地部署。
我们推出Gemma 3，是Gemma系列轻量级开源模型的新成员，参数规模从10亿到270亿不等。该版本新增视觉理解能力，覆盖更广的语言种类，并支持至少128K token的长上下文。通过增加局部注意力层比例并缩短局部注意力跨度，优化了模型架构，有效缓解长上下文下KV缓存爆炸问题。所有Gemma 3模型均采用蒸馏训练，预训练和指令微调版本性能均优于Gemma 2。尤其通过新设计的后训练流程，显著提升数学推理、对话交互、指令遵循和多语言能力。其中，Gemma3-4B-IT在多项基准上表现媲美Gemma2-27B-IT，而Gemma3-27B-IT性能接近Gemini-1.5-Pro。所有模型已向社区开源。
原文摘要 · Abstract (English)
We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。