让大模型懂通信领域，用8亿词元数据微调后性能逼近通用大模型。
TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain
- 收集800M词元、8万条指令的通信领域专用数据集
- 仅通过指令微调，小模型性能媲美大型通用模型
- 适合需要精准通信知识的工业场景应用
尽管大语言模型在诸多任务中表现优异，但在高度技术性领域仍缺乏准确性。通信领域尤为挑战，因其存在大量词汇、语义和概念上的特殊性。然而该领域蕴含众多直接关联产业需求的应用价值。本文研究如何将大语言模型适配至通信领域，报告了三项工作：(i) 收集大规模领域特定数据（8亿词元，8万条指令），(ii) 采用多种方法进行模型适配，(iii) 在需深度通信知识的下游任务中，对适配模型与更大规模通用模型进行基准测试。基于Llama-2-7b的实验表明，领域适配模型可与大型通用模型竞争。结果还显示，仅需一次指令微调即可实现有效适配，无需预先对原始文本进行微调。
原文摘要 · Abstract (English)
Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。