根据主题和学习表现动态调整数据权重，提升大模型预训练效果
ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selection
- 基于主题与学习模式动态重加权训练样本
- 多领域困惑度下降更快，下游任务性能更优
- 适合关注预训练数据优化的研究者与工程师
大规模语言模型的预训练需要海量多样化文本语料，高效的数据选择成为平衡计算资源与模型性能的关键挑战。现有方法多关注数据质量指标与混合比例，却未能充分捕捉样本间的语义关联及单个领域的质量差异。我们提出ToReMi（Topic-based Reweighting for Model improvement），一种两阶段框架，根据样本的主题关联性与观测到的学习模式动态调整其训练权重。全面实验表明，ToReMi变体在多个领域均优于传统预训练方法，实现更快的困惑度下降，并在下游评估任务中表现出更强能力。代码已公开于 https://github.com/zxx000728/ToReMi。
原文摘要 · Abstract (English)
Pre-training large language models (LLMs) necessitates enormous diverse textual corpora, making effective data selection a key challenge for balancing computational resources and model performance. Current methodologies primarily emphasize data quality metrics and mixing proportions, yet they fail to adequately capture the underlying semantic connections between training samples and quality disparities within individual domains. We introduce ToReMi (Topic-based Reweighting for Model improvement), a novel two-stage framework that dynamically adjusts training sample weights according to their topical associations and observed learning patterns. Our comprehensive experiments reveal that ToReMi variants consistently achieve superior performance over conventional pre-training approaches, demonstrating accelerated perplexity reduction across multiple domains and enhanced capabilities on downstream evaluation tasks. Code is available at https://github.com/zxx000728/ToReMi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。