双目标训练让语言模型既快又抗过拟合，性能超越单一目标模型。
Dual-objective Language Models: Training Efficiency Without Overfitting
- 同时使用自回归与掩码扩散目标，不改架构即可提升效果。
- 在50种数据重复率下测试，双目标始终最优且平衡点稳定。
- 适合追求高效训练且避免过拟合的NLP研究者与应用开发者。
本文在不修改模型架构的前提下，结合自回归与掩码扩散训练目标，构建出灵活的语言模型，其性能优于单一目标模型。自回归模型训练效率高，但易过拟合；掩码扩散模型抗过拟合能力强，但训练效率低。本研究通过在50种不同数据重复率下训练和评估语言模型，证明在所有测试设置下，双目标联合训练均表现最优，且最优平衡点在追求自回归或掩码扩散下游任务性能时保持一致。结果表明，双目标训练可兼顾训练效率与泛化能力。
原文摘要 · Abstract (English)
This paper combines autoregressive and masked-diffusion training objectives without any architectural modifications, resulting in flexible language models that outperform single-objective models. Autoregressive modeling has been a popular approach, partly because of its training efficiency; however, that comes at the cost of sensitivity to overfitting. On the other hand, masked-diffusion models are less efficient to train while being more resilient to overfitting. In this work, we demonstrate that dual-objective training achieves the best of both worlds. To derive the optimal balance between both objectives, we train and evaluate 50 language models under varying levels of data repetition. We show that it is optimal to combine both objectives under all evaluated settings and that the optimal balance is similar whether targeting autoregressive or masked-diffusion downstream performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。