用首个词概率优化检索，提升电信领域选择题回答准确率
First Token Probability Guided RAG for Telecom Question Answering
- 根据首个生成词的概率动态调整检索上下文和超参数
- 在电信多选题任务中显著降低幻觉，准确率优于基线方法
- 适合需要高精度知识问答的工业场景，如客服系统
大语言模型（LLMs）在通用能力上表现突出。针对需要复杂领域知识的应用，检索增强生成（RAG）通过引入领域特定信息展现出优势。然而，现有RAG研究尚未充分解决电信领域多选题问答（MCQA）中的检索质量与幻觉问题。为此，我们提出一种基于首个词概率引导的RAG框架。该框架利用置信度分数优化关键超参数，如块数和块窗口大小，并动态调整上下文。方法首先检索最相关段落，生成一个候选答案词；随后对所有选项的概率进行归一化，作为置信度分数，指导上下文动态调整。通过迭代优化超参数，持续提升RAG性能。实验验证了该框架的有效性，在领域特定多选题任务中表现出更高准确率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have garnered significant attention for their impressive general-purpose capabilities. For applications requiring intricate domain knowledge, Retrieval-Augmented Generation (RAG) has shown a distinct advantage in incorporating domain-specific information into LLMs. However, existing RAG research has not fully addressed the challenges of Multiple Choice Question Answering (MCQA) in telecommunications, particularly in terms of retrieval quality and mitigating hallucinations. To tackle these challenges, we propose a novel first token probability guided RAG framework. This framework leverages confidence scores to optimize key hyperparameters, such as chunk number and chunk window size, while dynamically adjusting the context. Our method starts by retrieving the most relevant chunks and generates a single token as the potential answer. The probabilities of all options are then normalized to serve as confidence scores, which guide the dynamic adjustment of the context. By iteratively optimizing the hyperparameters based on these confidence scores, we can continuously improve RAG performance. We conducted experiments to validate the effectiveness of our framework, demonstrating its potential to enhance accuracy in domain-specific MCQA tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。