arXiv:2601.03725cs.LG2026-01被引 2

动态调整训练数据顺序,让大模型学得更快更准。

EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuning

  • 根据模型推理时的不确定性动态选难样本训练。
  • 在医疗、法律等领域显著提升Qwen3-4B和Llama3.2-3B性能。
  • 计算效率提升83.5%,适合资源受限场景下的微调。

领域专用大语言模型通常通过在特定数据集上微调预训练通用模型实现,是应用AI的重要进展。当前主流的微调策略采用课程学习,按难度预排序训练样本以提高效率。但现有方法多使用静态课程,无法随模型状态动态调整。为此,本文提出EDCO框架,基于推理熵与动态课程编排两个核心思想:借鉴高回答熵有助于长期推理的发现,EDCO持续优先选择高推理熵的样本。该框架包含三部分:高效熵估计器(用前缀标记近似全序列熵)、基于熵的课程生成器(挑选最高熵数据点)、以及优化模型的LLM训练器。在通信、医学和法律领域对Qwen3-4B和Llama3.2-3B模型进行充分实验,EDCO在监督与强化学习设置下均优于传统课程策略。此外,所提高效熵估计使计算时间减少83.5%且保持高精度。

原文摘要 · Abstract (English)

Domain-specific large language models (LLMs), typically developed by fine-tuning a pre-trained general-purpose LLM on specialized datasets, represent a significant advancement in applied AI. A common strategy in LLM fine-tuning is curriculum learning, which pre-orders training samples based on metrics like difficulty to improve learning efficiency compared to a random sampling strategy. However, most existing methods for LLM fine-tuning rely on a static curriculum, designed prior to training, which lacks adaptability to the model's evolving needs during fine-tuning. To address this, we propose EDCO, a novel framework based on two key concepts: inference entropy and dynamic curriculum orchestration. Inspired by recent findings that maintaining high answer entropy benefits long-term reasoning gains, EDCO prioritizes samples with high inference entropy in a continuously adapted curriculum. EDCO integrates three core components: an efficient entropy estimator that uses prefix tokens to approximate full-sequence entropy, an entropy-based curriculum generator that selects data points with the highest inference entropy, and an LLM trainer that optimizes the model on the selected curriculum. Comprehensive experiments in communication, medicine and law domains, EDCO outperforms traditional curriculum strategies for fine-tuning Qwen3-4B and Llama3.2-3B models under supervised and reinforcement learning settings. Furthermore, the proposed efficient entropy estimation reduces computational time by 83.5% while maintaining high accuracy.

大模型微调课程学习推理熵动态优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。