70亿参数波兰语大模型，性能显著超越同类
Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation
- 采用加权指令交叉熵损失与自适应学习率优化训练
- 在RAG阅读任务上比Mistral-7B高9个百分点，推理与角色扮演表现突出
- 专为波兰语设计，适合本地化应用与语言研究者使用
我们推出Bielik 7B v0.1，一个70亿参数的波兰语生成文本模型。该模型基于精心筛选的波兰语语料库训练，通过创新技术解决语言模型开发中的关键挑战：包括加权指令交叉熵损失，以平衡不同指令类型的学習；以及自适应学习率，根据训练进度动态调整。为评估性能，我们构建了Open PL LLM Leaderboard和Polish MT-Bench两个新框架，用于评估各类NLP任务与对话能力。Bielik 7B v0.1表现出显著提升，在RAG阅读任务上平均得分比Mistral-7B-v0.1高出9个百分点；在Polish MT-Bench中，推理类得分为6.15/10，角色扮演类得分为7.83/10。该模型代表波兰语AI的重大进展，为多样化语言应用提供强大工具，并树立了新的基准。
原文摘要 · Abstract (English)
We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。