arXiv:2603.24580cs.CLcs.AI2026-03被引 2

RAG在政策问答中,检索增强未必提升答案可靠性。

Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

  • 用对比学习微调ColBERT检索器,结合偏好优化生成器
  • 更强检索反而导致无相关文档时更自信的幻觉
  • 适合关注政策问答系统可靠性的研究者与开发者

检索增强生成(RAG)系统被广泛用于分析复杂政策文件,但在法律语言密集、法规动态交叉的领域,仍难以满足专家级使用需求。本研究基于AI治理与监管档案(AGORA)语料库(947篇AI政策文档),构建了结合对比学习微调的ColBERT检索器与通过直接偏好优化(DPO)对齐人类偏好的生成器。通过合成查询和成对偏好数据,使系统适配政策领域。实验显示,领域微调提升了检索指标,但未稳定改善端到端问答性能;在无相关文档时,更强检索反而导致更自信的幻觉。这表明,组件优化不必然带来更可靠的问答结果。研究为动态法规语料上的可信赖问答系统设计提供了实用洞见。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) systems are increasingly used to analyze complex policy documents, but achieving sufficient reliability for expert usage remains challenging in domains characterized by dense legal language and evolving, overlapping regulatory frameworks. We study the application of RAG to AI governance and policy analysis using the AI Governance and Regulatory Archive (AGORA) corpus, a curated collection of 947 AI policy documents. Our system combines a ColBERT-based retriever fine-tuned with contrastive learning and a generator aligned to human preferences using Direct Preference Optimization (DPO). We construct synthetic queries and collect pairwise preferences to adapt the system to the policy domain. Through experiments evaluating retrieval quality, answer relevance, and faithfulness, we find that domain-specific fine-tuning improves retrieval metrics but does not consistently improve end-to-end question answering performance. In some cases, stronger retrieval counterintuitively leads to more confident hallucinations when relevant documents are absent from the corpus. These results highlight a key concern for those building policy-focused RAG systems: improvements to individual components do not necessarily translate to more reliable answers. Our findings provide practical insights for designing grounded question-answering systems over dynamic regulatory corpora.

RAG政策问答幻觉检索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。