打造中英双语金融大模型评测基准,全面评估模型能力
Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models
- 构建涵盖8项金融NLP任务的中英双语评测集
- 对比GPT-4o、Llama3等模型表现,揭示其优劣势
- 开源评测工具与Touchstone-GPT模型供研究使用
随着大语言模型在金融领域的深入应用,亟需一种标准化方法全面评估其性能。现有金融评测基准普遍存在语言和任务覆盖有限、数据质量不高、不适应LLM评估等问题。为此,我们提出Golden Touchstone,一个涵盖中英文8个核心金融NLP任务的综合性双语评测基准。该基准基于大量开源数据和行业需求构建,全面评估模型的语言理解与生成能力。通过对GPT-4o、Llama3、FinGPT、FinMA等主流模型的对比分析,揭示其在处理复杂金融信息时的优势与局限。此外,我们开源了通过持续预训练与指令微调训练的Touchstone-GPT金融大模型,其在双语评测中表现优异,但在特定任务上仍存不足。本研究为金融LLM提供实用评估工具,并指导未来优化方向。Golden Touchstone代码及Touchstone-GPT模型权重已公开于https://github.com/IDEA-FinAI/Golden-Touchstone。
原文摘要 · Abstract (English)
As large language models (LLMs) increasingly permeate the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. Existing financial benchmarks often suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for LLM evaluation. To address these limitations, we introduce Golden Touchstone, a comprehensive bilingual benchmark for financial LLMs, encompassing eight core financial NLP tasks in both Chinese and English. Developed from extensive open-source data collection and industry-specific demands, this benchmark thoroughly assesses models' language understanding and generation capabilities. Through comparative analysis of major models such as GPT-4o, Llama3, FinGPT, and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-source Touchstone-GPT, a financial LLM trained through continual pre-training and instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific tasks. This research provides a practical evaluation tool for financial LLMs and guides future development and optimization. The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at https://github.com/IDEA-FinAI/Golden-Touchstone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。