用聚类采样法，从通用数据中高效训练专用语言模型
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
- 通过聚类-重要性采样，从通用语料中选择贴近任务特性的数据
- 在多个领域上降低困惑度，提升多选题准确率
- 适合数据稀缺场景，尤其适用于多任务和持续预训练
专用语言模型（LMs）在特定任务或领域上表现优于同规模的通用模型，但多数任务缺乏足够的专用数据进行预训练。本文提出一种新方法——聚类重要性采样（CRISP），利用大规模通用数据集构建专用模型。该方法先对通用数据集进行聚类，再根据小规模专用数据集中各类别的频率，从聚类中采样。该方法可扩展，适用于预训练与持续预训练，且在多任务场景下表现良好。相比其他依赖有限领域数据调整通用数据分布的方法，CRISP更具优势。实验显示其在多个领域均降低了语言建模困惑度，并提升了多选题任务的准确率。消融实验分析了数据集大小、聚类配置和模型规模的影响。
原文摘要 · Abstract (English)
Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most tasks. In this work, we build specialist models from large generalist training sets instead. We propose a novel method, ClusteRed Importance SamPling (CRISP). CRISP clusters the generalist dataset and samples from these clusters based on their frequencies in the smaller specialist dataset. It is scalable, suitable for both pretraining and continued pretraining, and works well in multi-task settings. CRISP performs favorably compared to other methods that adjust the training distribution of the generalist data with guidance from the limited domain-specific data. Our findings demonstrate improvements across different domains in terms of language modeling perplexity and accuracy on multiple-choice question tasks. We also present ablation studies that examine the impact of dataset sizes, clustering configurations, and model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。