用印度注册会计师考试题测试大模型，发现顶尖模型在法律推理上表现好但算术仍有短板。
Large Language Models Acing Chartered Accountancy
- 构建针对注册会计师考试的专用测评集CA-Ben
- GPT-4o和Claude 3.5 Sonnet在法律推理中领先
- 算术计算与法律解读仍是当前大模型薄弱环节
大型语言模型（LLMs）正深刻改变金融实践，但其在特定金融领域知识的理解与应用能力尚不明确。为填补印度金融语境下的评估空白，本文提出CA-Ben基准，专用于评测模型在财务、法律及量化推理方面的能力。该数据集源自印度特许会计师学会（ICAI）的初、中、高级考试题，涵盖完整课程体系。六款主流模型——GPT-4o、LLAMA 3.3 70B、LLAMA 3.1 405B、MISTRAL Large、Claude 3.5 Sonnet、Microsoft Phi 4——在统一协议下被评估。结果显示，Claude 3.5 Sonnet与GPT-4o在概念与法律推理中表现最优；但在数值计算与法律条文理解方面仍存在明显不足。研究强调当前模型在定量分析与准确法律解释上的局限性，建议通过混合推理与检索增强生成技术加以改进。
原文摘要 · Abstract (English)
Advanced intelligent systems, particularly Large Language Models (LLMs), are significantly reshaping financial practices through advancements in Natural Language Processing (NLP). However, the extent to which these models effectively capture and apply domain-specific financial knowledge remains uncertain. Addressing a critical gap in the expansive Indian financial context, this paper introduces CA-Ben, a Chartered Accountancy benchmark specifically designed to evaluate the financial, legal, and quantitative reasoning capabilities of LLMs. CA-Ben comprises structured question-answer datasets derived from the rigorous examinations conducted by the Institute of Chartered Accountants of India (ICAI), spanning foundational, intermediate, and advanced CA curriculum stages. Six prominent LLMs i.e. GPT 4o, LLAMA 3.3 70B, LLAMA 3.1 405B, MISTRAL Large, Claude 3.5 Sonnet, and Microsoft Phi 4 were evaluated using standardized protocols. Results indicate variations in performance, with Claude 3.5 Sonnet and GPT-4o outperforming others, especially in conceptual and legal reasoning. Notable challenges emerged in numerical computations and legal interpretations. The findings emphasize the strengths and limitations of current LLMs, suggesting future improvements through hybrid reasoning and retrieval-augmented generation methods, particularly for quantitative analysis and accurate legal interpretation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。