探究温度与噪声对RAG系统鲁棒性的联合影响
TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness
- 设计框架，同时测试温度与文本扰动的交互作用
- 高温显著放大噪声敏感性，不同扰动类型响应非线性
- 提供评估、分析与调参指南，适合系统优化者
RAG系统的评估通常孤立考察检索质量与生成温度等参数，忽略其交互影响。本文系统研究了文本扰动（模拟噪声检索）与不同温度设置在多次LLM运行中的协同效应。提出一个全面的RAG扰动-温度分析框架，对HotpotQA数据集上的检索文档施加三种不同类型扰动，并在多种温度下测试。实验涵盖开源与专有LLM，结果表明：高温设置始终加剧对扰动的脆弱性；部分扰动类型在温度变化中呈现非线性敏感性。主要贡献包括：(1) 一套用于评估RAG鲁棒性的诊断基准；(2) 量化扰动-温度交互的分析框架；(3) 面对噪声检索时的模型选择与参数调优实用建议。
原文摘要 · Abstract (English)
The evaluation of Retrieval-Augmented Generation (RAG) systems typically examines retrieval quality and generation parameters like temperature in isolation, overlooking their interaction. This work presents a systematic investigation of how text perturbations (simulating noisy retrieval) interact with temperature settings across multiple LLM runs. We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings. Through extensive experiments on HotpotQA with both open-source and proprietary LLMs, we demonstrate that performance degradation follows distinct patterns: high-temperature settings consistently amplify vulnerability to perturbations, while certain perturbation types exhibit non-linear sensitivity across the temperature range. Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。