用新方法让小模型在通信问答中准确率提升近一倍。
QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
- 用问题掩码和选项重排优化训练，提升小模型理解力。
- 在通信多选题上，小模型准确率最高提升84.65%。
- 适合想低成本部署专业问答系统的开发者参考。
大语言模型在问答系统中取得显著进展，但在通信等专业领域因术语复杂、需精确回答而面临挑战。现有工作虽在RAG框架下用GPT-3.5取得较好效果，但受限于闭源与高算力需求。本文提出QMOS，通过问题掩码损失与选项重排策略，增强开源小型模型（Phi-2与Falcon-7B）在通信领域多选题上的表现。改进涵盖微调、检索、提示工程与推理全流程。实验显示，相较基线，使用Falcon-7B时准确率提升24.70%至49.30%，使用Phi-2时提升42.07%至84.65%。
原文摘要 · Abstract (English)
Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。