评测大模型在魁北克保险领域的表现,发现推理能力比参数更重要。
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
- 用链式思维推理的模型显著优于普通指令微调模型。
- 检索增强让知识弱的模型准确率提升超35个百分点,但可能引发灾难性错误。
- 通用大模型比小规模领域专用模型表现更好,适合高风险场景使用。
魁北克省保险数字化进程因《法案141》加速推进,导致消费者面临复杂的金融合同解读难题,形成‘建议缺口’。尽管大语言模型(LLMs)可提供可扩展的自动化咨询方案,但其在高风险领域的应用需确保法律准确性与可信度。本文提出AEPC-QA,一个基于官方监管认证手册的私有高质量基准数据集,包含807道多选题。我们对51个LLMs在闭书生成和检索增强生成(RAG)两种范式下进行了全面评估。结果揭示三大关键发现:1)推理时思维链(chain-of-thought)处理显著提升性能,如o3-2025-04-16、o1-2024-12-17等模型表现优异;2)RAG使参数知识薄弱模型准确率提升超35个百分点,却导致部分模型出现‘上下文干扰’,引发严重性能下降;3)存在‘专业化悖论’,大型通用模型持续优于小型法语领域微调模型。当前架构已接近专家水平(约79%),但外部检索引入的不稳定性要求部署前必须进行严格鲁棒性校准。
原文摘要 · Abstract (English)
The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。