用少量数据训练出比大模型更强的波斯语医学推理模型
Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training
- 用AI反馈强化学习生成思维链对错样本
- 200万词元正样本使小模型超越5700万词元大模型
- 适合资源有限的低资源语言医疗AI开发
在波斯语等低资源语言中，提升小型语言模型的推理能力对医疗问答等专业应用至关重要。本研究采用基于AI反馈的强化学习（RLAIF）与直接偏好优化（DPO），改进通用波斯语模型的医学推理能力。我们翻译了多选题医学问答数据集，并利用RLAIF生成正负答案对，通过提示教师与学生模型生成思维链（CoT）推理过程，构建包含正确与错误推理路径的数据集。该数据集包含200万词元的优选回答和250万词元的拒选回答，用于训练基线模型，显著提升了其波斯语医学推理能力。令人惊讶的是，该模型性能超越了使用约5700万词元训练的gaokerena-V模型，凸显了聚焦推理训练在数据受限场景下的高效性与有效性。
原文摘要 · Abstract (English)
Enhancing reasoning capabilities in small language models is critical for specialized applications such as medical question answering, particularly in underrepresented languages like Persian. In this study, we employ Reinforcement Learning with AI Feedback (RLAIF) and Direct preference optimization (DPO) to improve the reasoning skills of a general-purpose Persian language model. To achieve this, we translated a multiple-choice medical question-answering dataset into Persian and used RLAIF to generate rejected-preferred answer pairs, which are essential for DPO training. By prompting both teacher and student models to produce Chain-of-Thought (CoT) reasoning responses, we compiled a dataset containing correct and incorrect reasoning trajectories. This dataset, comprising 2 million tokens in preferred answers and 2.5 million tokens in rejected ones, was used to train a baseline model, significantly enhancing its medical reasoning capabilities in Persian. Remarkably, the resulting model outperformed its predecessor, gaokerena-V, which was trained on approximately 57 million tokens, despite leveraging a much smaller dataset. These results highlight the efficiency and effectiveness of reasoning-focused training approaches in developing domain-specific language models with limited data availability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。