用人类标注的简单语言构建课程，能有效提升语言模型预训练效果。
Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning
- 基于维基百科简单版文章标签设计课程顺序
- 先学简单文本可显著降低简单语言的困惑度
- 人工判断的语言难度比浅层启发式更适合作为课程信号
课程学习（CL）旨在通过从“简单”到“复杂”的顺序训练数据来提升模型性能，但如何定义和衡量语言难度仍是开放问题。本文研究人类标注的简单语言是否可作为有效的课程信号。利用 Simple Wikipedia 数据集的文章级标签，我们对比了基于标签的课程策略与依赖浅层启发式的基于能力的策略。实验采用 BERT-tiny 模型，结果表明：单纯加入简单数据并无明显收益；但若以课程形式组织，尤其是优先引入简单数据，能持续降低困惑度，尤其在简单语言任务上表现突出。相比之下，基于能力的课程未带来稳定提升，可能因无法有效区分两类数据。结果表明，人类对语言难度的直觉可有效指导语言模型预训练中的课程学习。
原文摘要 · Abstract (English)
Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。