改进扩散语言模型的训练策略，提升数学与逻辑推理能力
DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns
- 通过调整掩码策略和损失函数，引导模型理解数学与逻辑模式
- 在小数据上使数学题准确率提升5-10%，逻辑题提升约2%
- 方法简单易用，可融合多种训练方式，适用于各类扩散语言模型
扩散大语言模型（dLLMs）作为自回归模型之后的新架构，其去噪过程具备强大生成能力，但在学习数值敏感的数学任务和顺序敏感的逻辑任务时仍面临挑战。现有预训练、微调和强化学习等方法主要提升通用知识与推理能力，但缺乏对数学与逻辑模式的深入理解。本文提出DSFT——一种简单有效的扩散监督微调策略，通过优化掩码机制与损失函数，引导模型更好地掌握数学与逻辑模式。该策略可灵活结合预训练、强化学习等多种训练方法。在LLaDA与Dream系列模型上的实验表明，仅使用小规模数据，即可在数学问题上实现5%-10%的性能提升，在逻辑问题上提升约2%。该掩码设计为特定模式学习提供了新思路，且易于集成至其他训练流程，可广泛应用于各类dLLM。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) have emerged as a new architecture following auto regressive models. Their denoising process offers a powerful generative advantage, but they present significant challenges in learning and understanding numerically sensitive mathematical and order-sensitive logical tasks. Current training methods, including pre-training, fine-tuning, and reinforcement learning, focus primarily on improving general knowledge retention and reasoning abilities, but lack a comprehensive understanding of mathematical and logical patterns. We propose DSFT, a simple yet effective Diffusion SFT strategy, by adjusting the masking strategy and loss function, guiding models to understand mathematical and logical patterns. This strategy can be flexibly combined with pre-training, reinforcement learning, and other training methods. Validated on models such as LLaDA and Dream series, we prove that DSFT on small-scale data can achieve improvements of 5-10% and approximately 2% on mathematical and logical problems, respectively. This inspiring masking approach offers insights for future learning of specific patterns, which can be easily and efficiently combined with other training methods and applied to various dLLMs. Our code is publicly available at https://anonymous.4open.science/r/DSFT-0FFB/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。