让大模型更关注难题，提升自训练效果
DAST: Difficulty-Aware Self-Training on Large Language Models
- 根据难度动态采样，识别并聚焦难问题
- 通过难度感知增强数据，提升难例学习质量
- 适用于数学等需要深度推理的任务
当前的大语言模型自训练方法往往低估了难题的重要性，导致对复杂问题的学习不足。为此，本文提出一种难度感知的自训练框架（DAST），旨在提升自生成回答在难题上的数量与质量。DAST包含三个核心组件：基于采样的难度等级估计、难度感知的数据增强，以及分别结合监督微调（SFT）和直接偏好优化（DPO）的自训练算法。在数学任务上的实验表明，DAST显著提升了模型性能，验证了难度感知策略在推进大模型自训练中的关键作用。
原文摘要 · Abstract (English)
Present Large Language Models (LLM) self-training methods always under-sample on challenging queries, leading to inadequate learning on difficult problems which limits LLMs' ability. Therefore, this work proposes a difficulty-aware self-training (DAST) framework that focuses on improving both the quantity and quality of self-generated responses on challenging queries during self-training. DAST is specified in three components: 1) sampling-based difficulty level estimation, 2) difficulty-aware data augmentation, and 3) the self-training algorithm using SFT and DPO respectively. Experiments on mathematical tasks demonstrate the effectiveness and generalization of DAST, highlighting the critical role of difficulty-aware strategies in advancing LLM self-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。