测试AI在神学与牧灵指导中的表现,发现结构化引导能显著提升安全判断能力。
When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models

- 构建120个基督教神学情境的评估基准,检验LLM在信仰指导中的表现
- 结构化引导使AI在紧急情况识别上提升10.8分,平均得分提高3.96分
- 适合研究伦理、宗教AI应用或人机协同牧灵的学者与开发者
越来越多用户向大型语言模型(LLMs)寻求关于信仰、教义和牧灵关怀的建议。这些问题不同于普通信息查询:有的涉及核心基督教信念,有的关乎各传统间的分歧,有的需要谦卑态度因属实践性议题,还有的在牧灵情境中安全与人类转介比神学完备更重要。现有基准无法评估此类复杂性。本文提出FMG-Bench——一个包含120个场景的英文基督教神学分诊与牧灵指导基准,用于评估14个先进模型在8,792次评分响应中的表现。对比原始模型行为与三种引导指令设置,结果显示:引入结构化框架后,所有模型平均得分提升+3.96分;最关键的是,在需紧急转介的场景中,系统识别恰当性提升+10.8分。引导设置也增强了鲁棒性,重述或施压下稳定性从92.88升至98.02。在次要教义问题中,要求比较观点有帮助,但在核心教义或紧急牧灵情境中可能适得其反。该基准为评估工具,不意味着认可AI作为牧者权威。
原文摘要 · Abstract (English)
People increasingly ask large language models (LLMs) for counsel on questions of faith, doctrine, and pastoral care. These questions are not ordinary information requests. Some ask about core Christian beliefs, some ask about real disagreements among faithful traditions, some require humility because the issue is prudential, and some are pastoral situations where safety and human referral matter more than theological completeness. Existing benchmarks do not evaluate this structure. We introduce FMG-Bench, the Faith & Moral Guidance Benchmark, a 120-scenario benchmark for evaluating large language model behavior in English-language Christian theological triage and pastoral guidance contexts. FMG-Bench v1 evaluates 14 advanced models across 8,792 scored responses, comparing raw model behavior with three guided instruction settings. In our production run, placing models inside a structured harness improves over raw model behavior by +3.96 points on average, with every model improving. The most safety-critical finding is a +10.8 point gain in escalation appropriateness -- whether AI systems recognize when pastoral, clinical, legal, or emergency support is needed. The guided settings also improve robustness, meaning consistency when questions are reworded or pressured (92.88 to 98.02 stability). Asking a model to compare perspectives helps in secondary-doctrine questions but can be counterproductive when applied to primary doctrine or urgent pastoral situations. The benchmark is a measurement tool, not an endorsement of AI systems as pastoral authorities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。