根据任务需求动态调整掩码策略，提升文本模型下游性能。
Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
- 基于任务知识选择掩码词，区分有用与有害词汇。
- 采用周期性递减掩码率，实现从难到易的反课程学习。
- 在情感分析等3项任务上显著提升效果，适合改进预训练模型。
掩码语言建模已成为大规模语言模型预训练的主流无监督方法。然而，传统方法随机选择待掩码词，且掩码比例在整个训练过程中固定不变。本文提出一种任务感知的反课程掩码学习框架（TIACBM），通过任务特定知识判断哪些词应被掩码，并设计周期性递减的掩码比例，形成由难到易的反课程训练过程。我们在情感分析、主题分类和作者归属三个不同下游任务上验证该方法，结果表明，TIACBM能有效增强模型对关键任务特征的关注能力，在多个任务上取得统计显著的性能提升。代码已开源：https://github.com/JarcaAndrei/TIACBM。
原文摘要 · Abstract (English)
Masked language modeling has become a widely adopted unsupervised technique to pre-train large language models (LLMs). However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically fixed for the entire training process. In this paper, we propose to adjust the masking ratio and to decide which tokens to mask based on a novel task-informed anti-curriculum learning scheme. First, we harness task-specific knowledge about useful and harmful tokens in order to determine which tokens to mask. Second, we propose a cyclic decaying masking ratio, which corresponds to an anti-curriculum schedule (from hard to easy). We exemplify our novel task-informed anti-curriculum by masking (TIACBM) approach across three diverse downstream tasks: sentiment analysis, text classification by topic, and authorship attribution. Our findings suggest that TIACBM enhances the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. We release our code at https://github.com/JarcaAndrei/TIACBM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。