用多智能体RAG系统帮学生精准查校规,拒答敏感请求
Carolina Guide: A Multi-Agent RAG System with Institutional Guardrails for Academic Policy Assistance

- 构建多智能体RAG管道,结合机构规则约束答案输出
- 98.9%查询在前1位召回相关段落,安全拒答率86%
- 适合高校政策助手开发,强调安全与透明性
大学生常因难以理解复杂的学术政策而遭遇指导瓶颈,信息获取延迟。尽管大语言模型(LLMs)在自动化辅助方面有潜力,但其易产生幻觉且无法强制执行机构约束,缺乏安全设计则不适合高风险政策指导。我们提出 Carolina Guide,一个为南卡罗来纳大学(USC)设计的检索增强生成(RAG)系统,用于学术政策协助。该系统采用模块化多智能体架构,配备机构级安全守卫,提供有引用支持、基于政策的答案,并拒绝如课程推荐等不安全请求。在覆盖6个院系的90个查询测试集上,≥2个相关结果的召回率达98.9%,前1个相关段落的排名准确率同样为98.9%(MRR at 10 for rel ≥ 2 = 0.989)。通过系统基线对比和消融实验,验证了MMR重排序、充分检索上下文(k=20)及引用强制机制均具可衡量的实际价值。对30个对抗性查询的安全评估显示,安全F1为0.89,正确拒绝86%的不安全请求,同时保持93%良性查询覆盖率。结果表明,面向机构政策指导的生产级LLM系统需重构标准RAG模式,优先保障安全、透明与院系自主权,而非追求对话流畅性。
原文摘要 · Abstract (English)
University students often struggle to navigate complex academic policies, leading to advising bottlenecks and delayed access to critical information. Although large language models (LLMs) offer promise for automated assistance, their tendency toward hallucination and inability to enforce institutional constraints make them unsuitable for high-stakes policy guidance without careful architectural design. We present Carolina Guide, a retrieval-augmented generation (RAG) system for academic policy assistance at the University of South Carolina (USC). The system employs a modular multi-agent pipeline with institutional guardrails to provide citation-supported, policy-grounded answers to student queries while refusing unsafe requests such as course recommendations or personalized advising. We evaluate the system on a 90 query test set across 6 departments, achieving 98.9% retrieval success at the >= 2 threshold (genuinely relevant results) with the first relevant chunk at rank-1 for 98.9% of queries (MRR at 10 for rel >= 2 = 0.989). Through systematic baseline comparisons and ablation studies, we show that each architectural component-MMR reranking, adequate retrieval context (k=20), and citation enforcement-contributes measurable practical value despite limited statistical power at 90 queries. The evaluation of the guardrail on 30 adversarial queries demonstrates Safety F1 of 0.89, correctly refusing 86% of unsafe queries while maintaining 93% coverage of benign queries. These results show that production-ready LLM systems for institutional policy guidance require rethinking standard RAG patterns to prioritize safety, transparency, and departmental autonomy over conversational sophistication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。