研究发现用户语言变化会让RAG系统性能下降超40%,暴露其实际应用脆弱性。
Out of Style: RAG's Fragility to Linguistic Variation
- 测试四种语言特征对RAG影响,发现非正式和语法错误显著降低效果
- 最差情况下游召回率下降40.41%,答案匹配率下降38.86%
- 揭示RAG比纯大模型更易受语言变化干扰,适合关注真实场景鲁棒性的研究者
尽管检索增强生成(RAG)系统在多种自然语言处理基准上表现优异,但其在真实用户-大模型交互查询中的鲁棒性仍缺乏充分探索。这一差距严重影响实际部署,因用户查询存在更多语言变体,可能引发依赖组件间的级联错误。本文系统分析了正式度、可读性、礼貌性和语法正确性四个语言维度对RAG性能的影响。评估了两种检索模型与九个参数量从3亿到72亿不等的大语言模型,在四个信息获取型问答数据集上的表现。结果表明,语言改写显著影响检索与生成阶段,非正式查询导致Recall@5下降高达40.41%,含语法错误的查询使答案匹配率下降38.86%。值得注意的是,RAG系统对语言变化的敏感性高于仅使用大模型的生成方式,凸显其因语言波动导致错误传播的脆弱性。研究呼吁加强鲁棒性技术以提升多样化用户交互下的可靠性。代码已开源:https://github.com/Springcty/RAG-fragility-to-linguistic-variation。
原文摘要 · Abstract (English)
Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queries remains largely underexplored. This presents a critical gap for practical deployment, where user queries exhibit greater linguistic variations and can trigger cascading errors across interdependent RAG components. In this work, we systematically analyze how varying four linguistic dimensions (formality, readability, politeness, and grammatical correctness) impact RAG performance. We evaluate two retrieval models and nine LLMs, ranging from 3 to 72 billion parameters, across four information-seeking Question Answering (QA) datasets. Our results reveal that linguistic reformulations significantly impact both retrieval and generation stages, leading to a relative performance drop of up to 40.41% in Recall@5 scores for less formal queries and 38.86% in answer match scores for queries containing grammatical errors. Notably, RAG systems exhibit greater sensitivity to such variations compared to LLM-only generations, highlighting their vulnerability to error propagation due to linguistic shifts. These findings highlight the need for improved robustness techniques to enhance reliability in diverse user interactions. Code is available at https://github.com/Springcty/RAG-fragility-to-linguistic-variation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。