通过不确定性感知目标提升语言模型后训练效果
Token-Level Uncertainty-Aware Objective for Language Model Post-Training
- 结合掩码最大似然与自蒸馏，实现分词级不确定性建模
- 在多个模型和数据集上显著降低过拟合，提升分布外性能
- 适合需要稳定微调的场景，尤其对小样本任务有效
本文将因果语言建模中的分词级不确定性与两类训练目标关联：1）掩码最大似然（MLE），2）自蒸馏。结果表明，掩码 MLE 能有效降低认知不确定性，并作为自动课程学习机制；但易过拟合，需依赖自蒸馏正则化以维持或提升分布外任务表现。所提出的联合训练目标（掩码 MLE + 自蒸馏）在多种架构（Gemma、LLaMA、Phi）和数据集（Alpaca、ShareGPT、GSM8K）上均实现显著性能提升，在抑制过拟合的同时保持后训练过程的适应性。研究揭示了不确定性感知训练对增强语言模型的有效性。
原文摘要 · Abstract (English)
In the current work, we connect token-level uncertainty in causal language modeling to two types of training objectives: 1) masked maximum likelihood (MLE), 2) self-distillation. We show that masked MLE is effective in reducing epistemic uncertainty, and serve as an effective token-level automatic curriculum learning technique. However, masked MLE is prone to overfitting and requires self-distillation regularization to improve or maintain performance on out-of-distribution tasks. We demonstrate significant performance gain via the proposed training objective - combined masked MLE and self-distillation - across multiple architectures (Gemma, LLaMA, Phi) and datasets (Alpaca, ShareGPT, GSM8K), mitigating overfitting while maintaining adaptability during post-training. Our findings suggest that uncertainty-aware training provides an effective mechanism for enhancing language model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。