通过筛选高信息量词元，让大模型预训练更高效且鲁棒。
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
- 基于词元熵或损失值动态选择每批最有价值的词元
- 减少40%以上计算量，同时保持甚至提升困惑度与下游性能
- 适合追求高效预训练的模型开发者和资源受限场景
大语言模型预训练计算开销巨大，但许多词元对学习贡献有限。本文提出高效选择性语言建模（ESLM），一种风险感知算法，通过在线词元级批次筛选提升训练效率与分布鲁棒性。ESLM利用词元级统计量（如熵或损失），采用风险价值阈值筛选每批中最具信息量的词元，重塑训练损失，优先处理高风险词元并消除冗余梯度计算。我们将ESLM建模为双层博弈：模型与一个在约束阈值下选择最坏情况词元子集的掩码对手对抗。在损失导向设定中，ESLM恢复条件风险价值损失最小化，建立了与分布鲁棒优化的理论联系。我们进一步提出Ada-ESLM，可在训练中自适应调整选择置信度。GPT-2预训练实验表明，ESLM显著降低训练浮点运算量，同时维持或优于基线的困惑度与下游性能。该方法在不同模型规模、预训练语料上均具可扩展性，并可自然融合知识蒸馏。
原文摘要 · Abstract (English)
Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce Efficient Selective Language Modeling (ESLM), a risk-aware algorithm that improves training efficiency and distributional robustness by performing online token-level batch selection. ESLM leverages per-token statistics (e.g., entropy or loss) and applies value-at-risk thresholding to retain only the most informative tokens per batch. This data-centric mechanism reshapes the training loss, prioritizing high-risk tokens and eliminating redundant gradient computation. We frame ESLM as a bilevel game: the model competes with a masking adversary that selects worst-case token subsets under a constrained thresholding rule. In the loss-based setting, ESLM recovers conditional value-at-risk loss minimization, providing a principled connection to distributionally robust optimization. We extend our approach to Ada-ESLM, which adaptively tunes the selection confidence during training. Experiments on GPT-2 pretraining show that ESLM significantly reduces training FLOPs while maintaining or improving both perplexity and downstream performance compared to baselines. Our approach also scales across model sizes, pretraining corpora, and integrates naturally with knowledge distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。