开源大模型Llama-3-Motif专攻韩语，性能媲美GPT-4。
Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
- 基于Llama 3架构，用新训练技术扩展韩语能力
- 1020亿参数，在韩语评测中超越现有模型
- 适合需要高精度韩语生成的开发者与研究者
我们提出Llama-3-Motif，一个拥有1020亿参数的语言模型，专门增强韩语能力同时保持英文表现。该模型基于Llama 3架构，采用LlamaPro和掩码结构生长等先进训练技术，在不改变核心Transformer结构的前提下实现高效扩展。利用MoAI平台在超大规模GPU集群上训练，通过精心筛选的数据集（韩英数据比例均衡）优化模型。Llama-3-Motif在韩语专用基准测试中表现良好，优于现有模型，性能接近GPT-4。
原文摘要 · Abstract (English)
We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English. Developed on the Llama 3 architecture, Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture. Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data. Llama-3-Motif shows decent performance on Korean-specific benchmarks, outperforming existing models and achieving results comparable to GPT-4.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。