8B模型专精英语水平测评生成，效果优于GPT-3.5
\llinstruct: An Instruction-tuned model for English Language Proficiency Assessments
- 基于7万条测评指令微调Llama-3 8B模型
- 70K数据集训练的模型输出最符合评估要求
- 适合教育测评系统开发人员参考
我们提出\ llinstruct：一个80亿参数的指令微调模型，专为英语语言能力评估（ELPA）及相关应用生成内容。研究构建了一个包含7万条指令与解释的新数据集，并用其对Llama-3 8B模型（如SFT-17K、SFT-50K和SFT-70K）进行微调。通过人类评估对比未见指令下的表现，发现所有SFT模型表现相当，但训练于最大数据集（SFT-70K）的模型生成结果最具有效性，可直接用于评估。尽管其解释质量优于GPT-3.5等大模型，多数输出仍需人工干预才能投入实际测评使用。
原文摘要 · Abstract (English)
We present \llinstruct: An 8B instruction-tuned model that is designed to generate content for English Language Proficiency Assessments (ELPA) and related applications. Our work involves creating a new dataset of 70K instructions and explanations in the ELPA domain and using these to fine-tune Llama-3 8B models (SFT) of different sizes (e.g., SFT-17K, SFT-50K and SFT-70K). Human evaluations are conducted over unseen instructions to compare these SFT models against SOTA models (e.g., Dolly-2, Mistral, Llama-3 base version, and GPT-3.5). The findings show although all three SFT models perform comparably, the model trained on largest instruction dataset -- SFT-70K - leads to the most valid outputs ready for assessments. However, although the SFT models perform better than larger model, e.g., GPT 3.5 on the aspect of explanations of outputs, many outputs still need human interventions to make them actual ready for real world assessments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。