用低成本方法从Common Crawl提取低资源语言数据，提升大模型表现
UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages
- 从全量Common Crawl中筛选低资源语言文本，仅需少量算力
- 微调后语言建模困惑度下降，少样本提示得分显著提升
- 适合在消费级硬件上改进低资源语言模型的开发者
大型语言模型在低资源语言上表现不佳，主要因训练数据不足。本文提出UnifiedCrawl方法，高效从整个Common Crawl语料库中收集低资源语言文本。该方法仅需极少计算资源，即可生成远超以往来源的单语数据集。实验表明，利用这些数据通过QLoRA等高效适配器微调多语言大模型，能显著提升低资源语言性能，同时大幅降低显存占用。结果显示语言建模困惑度明显下降，少样本提示任务准确率提升。本工作及开源代码为使用消费级硬件改进低资源语言大模型提供了经济可行方案。源代码已公开于https://github.com/bethelmelesse/unifiedcrawl。
原文摘要 · Abstract (English)
Large language models (LLMs) under-perform on low-resource languages due to limited training data. We present a method to efficiently collect text data for low-resource languages from the entire Common Crawl corpus. Our approach, UnifiedCrawl, filters and extracts common crawl using minimal compute resources, yielding mono-lingual datasets much larger than previously available sources. We demonstrate that leveraging this data to fine-tuning multilingual LLMs via efficient adapter methods (QLoRA) significantly boosts performance on the low-resource language, while minimizing VRAM usage. Our experiments show large improvements in language modeling perplexity and an increase in few-shot prompting scores. Our work and released source code provide an affordable approach to improve LLMs for low-resource languages using consumer hardware. Our source code is available here at https://github.com/bethelmelesse/unifiedcrawl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。