打造开源双语模型，显著提升韩语能力同时保持英语表现
DNA 1.0 Technical Report
- 在Llama 3.1基础上持续预训练并微调，强化韩语理解与生成
- 韩语任务表现领先：KMMLU 53.26%，KoBEST 83.40%，BELEBELE 57.99%
- 适合需要高精度韩英双语处理的研究与商业应用
本文介绍DNA 1.0 8B Instruct，一个面向韩英双语任务的先进语言模型。通过使用高质量韩语文本对Llama 3.1 8B进行持续预训练（CPT），并进行监督微调（SFT），构建出具备指令遵循能力的模型。随后通过球面线性插值（SLERP）与原Llama 3.1 8B Instruct融合，并经由直接偏好优化（DPO）与知识蒸馏（KD）进一步优化。该模型在韩语专属任务上达到顶尖水平：KMMLU 53.26%、KoBEST 83.40%、BELEBELE 57.99%，同时在英语任务中保持优异性能：MMLU 66.64%、MMLU-Pro 43.05%、GSM8K 80.52%。作为开源模型，DNA 1.0 8B Instruct 已在Hugging Face发布，网址：https://huggingface.co/dnotitia/Llama-DNA-1.0-8B-Instruct。商业授权或反馈请联系：https://www.dnotitia.com/contact/post-form。
原文摘要 · Abstract (English)
In this report, we present DNA 1.0 8B Instruct, a state-of-the-art bilingual language model optimized for Korean and English language tasks. By applying continual pre-training (CPT) with high-quality Korean datasets to Llama 3.1 8B and subsequent supervised fine-tuning (SFT), we create an instruction-following model with enhanced Korean language capabilities. This model is then merged with Llama 3.1 8B Instruct via spherical linear interpolation (SLERP) and undergoes further optimization through direct preference optimization (DPO) and knowledge distillation (KD). DNA 1.0 8B Instruct achieves state-of-the-art results on Korean-specific tasks, including KMMLU (53.26%), KoBEST (83.40%), and BELEBELE (57.99%), while maintaining strong English capabilities on MMLU (66.64%), MMLU-Pro (43.05%) and GSM8K (80.52%). As an open model, DNA 1.0 8B Instruct represents a significant advancement in bilingual language modeling. As an open model, DNA 1.0 8B Instruct is freely available through https://huggingface.co/dnotitia/Llama-DNA-1.0-8B-Instruct . For commercial licensing inquiries or feedback, please contact us at https://www.dnotitia.com/contact/post-form
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。