测试检索增强模型如何适应不同用户需求，发现限制记忆能提升鲁棒性。
Do Retrieval-Augmented Language Models Adapt to Varying User Needs?
- 设计三种用户需求场景，评估模型在不同检索条件下的表现。
- 理想检索下性能最优，但限制记忆可增强对抗性环境下的稳定性。
- 适用于关注模型真实应用适应性的研究者与开发者。
近年来，检索增强语言模型（RALMs）在知识密集型任务中展现出优异效果。然而，现有评测基准通常假设单一最优信息利用方式，未能反映用户需求的多样性。本文提出一种新评测框架，系统评估RALMs在三种用户需求场景——仅依赖上下文、优先使用上下文、优先依赖记忆——及三种上下文设置（上下文匹配、知识冲突、信息无关）下的表现。通过在HotpotQA、DisentQA及新构建的合成数据集URAQ上进行大量实验，发现限制记忆使用虽降低理想检索下的峰值性能，但在对抗性检索条件下显著提升鲁棒性，且模型家族差异主导行为变化。结果强调了用户中心化评测的重要性，并为优化模型在多变检索环境中的表现提供洞见。代码与URAQ数据集将在论文接收后开源。
原文摘要 · Abstract (English)
Recent advancements in Retrieval-Augmented Language Models (RALMs) have demonstrated their efficacy in knowledge-intensive tasks. However, existing evaluation benchmarks often assume a single optimal approach to leveraging retrieved information, failing to account for varying user needs. This paper introduces a novel evaluation framework that systematically assesses RALMs under three user need cases-Context-Exclusive, Context-First, and Memory-First-across three distinct context settings: Context Matching, Knowledge Conflict, and Information Irrelevant. By varying both user instructions and the nature of retrieved information, our approach captures the complexities of real-world applications where models must adapt to diverse user requirements. Through extensive experiments on multiple QA datasets, including HotpotQA, DisentQA, and our newly constructed synthetic URAQ dataset, we find that restricting memory usage improves robustness in adversarial retrieval conditions but decreases peak performance with ideal retrieval results and model family dominates behavioral differences. Our findings highlight the necessity of user-centric evaluations in the development of retrieval-augmented systems and provide insights into optimizing model performance across varied retrieval contexts. We will release our code and URAQ dataset upon acceptance of the paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。