针对国企领域大模型训练,提出全流程优化方案提升性能与推理速度。
SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding
- 通过持续预训练+渐进式微调+蒸馏加速推理,构建专用大模型
- 在保持99.8%通用能力下,领域指标提升1.08倍(Rouge-1)
- 采用72B→7B蒸馏推理,实现1.39-1.52倍加速且无质量损失
本研究针对中国国有企业资产与企业领域大语言模型(LLMs)开发中的三大挑战:模型容量受限、过度依赖领域标注数据、长文本推理效率低。提出SOAEsV2-7B/72B系列模型,采用三阶段框架:1)持续预训练在融入领域知识的同时保留基础能力;2)基于课程学习的渐进式监督微调(SFT),从弱相关对话数据逐步过渡到专家标注的国企数据集;3)基于蒸馏的推测解码,在72B目标模型与7B草稿模型间进行对数蒸馏,实现1.39–1.52×推理加速且无质量损失。实验表明,领域预训练阶段维持99.8%原始通用语言能力,同时使Rouge-1提升1.08×,BLEU-4提升1.17×。消融实验显示渐进式SFT优于单阶段训练,分别带来1.02×和1.06×的指标提升。该工作提供了面向国企领域的完整优化路径,弥合通用能力与领域专精之间的差距。
原文摘要 · Abstract (English)
This study addresses key challenges in developing domain-specific large language models (LLMs) for Chinese state-owned assets and enterprises (SOAEs), where current approaches face three limitations: 1) constrained model capacity that limits knowledge integration and cross-task adaptability; 2) excessive reliance on domain-specific supervised fine-tuning (SFT) data, which neglects the broader applicability of general language patterns; and 3) inefficient inference acceleration for large models processing long contexts. In this work, we propose SOAEsV2-7B/72B, a specialized LLM series developed via a three-phase framework: 1) continual pre-training integrates domain knowledge while retaining base capabilities; 2) domain-progressive SFT employs curriculum-based learning strategy, transitioning from weakly relevant conversational data to expert-annotated SOAEs datasets to optimize domain-specific tasks; 3) distillation-enhanced speculative decoding accelerates inference via logit distillation between 72B target and 7B draft models, achieving 1.39-1.52$\times$ speedup without quality loss. Experimental results demonstrate that our domain-specific pre-training phase maintains 99.8% of original general language capabilities while significantly improving domain performance, resulting in a 1.08$\times$ improvement in Rouge-1 score and a 1.17$\times$ enhancement in BLEU-4 score. Ablation studies further show that domain-progressive SFT outperforms single-stage training, achieving 1.02$\times$ improvement in Rouge-1 and 1.06$\times$ in BLEU-4. Our work introduces a comprehensive, full-pipeline approach for optimizing SOAEs LLMs, bridging the gap between general language capabilities and domain-specific expertise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。