专为通信领域打造的系列大模型，解决通用模型术语理解差的问题。
Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications
- 构建通信领域专用数据集Tele-Data与Tele-Eval，支持模型训练。
- 1B到8B参数模型在通信任务上优于通用模型，且不遗忘原有能力。
- 适合通信研究、工程开发人员使用，提升技术文档理解效率。
大型语言模型（LLMs）在多个领域广泛应用，但其在通信领域的应用仍有限，主要依赖缺乏专业性的通用模型，导致在处理通信术语及数学表达时表现不佳。本文首先构建了来自相关来源的通信材料数据集Tele-Data，以及针对该领域的大型问答数据集Tele-Eval。通过大量实验，探索了将LLM适配至通信领域的最佳训练方法，涵盖不同通信方向的专业化划分、参数高效微调技术，并研究了模型规模对适应性的影响及训练数据的作用。基于这些发现，我们开发并开源了首个面向通信领域的系列语言模型——Tele-LLMs，参数范围从1B到8B。评估表明，这些模型在Tele-Eval和通信文献任务上优于通用模型，同时保留了原有的通用能力，避免了灾难性遗忘。
原文摘要 · Abstract (English)
The emergence of large language models (LLMs) has significantly impacted various fields, from natural language processing to sectors like medicine and finance. However, despite their rapid proliferation, the applications of LLMs in telecommunications remain limited, often relying on general-purpose models that lack domain-specific specialization. This lack of specialization results in underperformance, particularly when dealing with telecommunications-specific technical terminology and their associated mathematical representations. This paper addresses this gap by first creating and disseminating Tele-Data, a comprehensive dataset of telecommunications material curated from relevant sources, and Tele-Eval, a large-scale question-and-answer dataset tailored to the domain. Through extensive experiments, we explore the most effective training techniques for adapting LLMs to the telecommunications domain, ranging from examining the division of expertise across various telecommunications aspects to employing parameter-efficient techniques. We also investigate how models of different sizes behave during adaptation and analyze the impact of their training data on this behavior. Leveraging these findings, we develop and open-source Tele-LLMs, the first series of language models ranging from 1B to 8B parameters, specifically tailored for telecommunications. Our evaluations demonstrate that these models outperform their general-purpose counterparts on Tele-Eval and telecommunications-related literature tasks while retaining their previously acquired capabilities, thus avoiding the catastrophic forgetting phenomenon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。