构建多语言多领域金融会议理解数据集，评测大模型真实场景理解力。
M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset
- 覆盖中英日三语、多个GICS行业，模拟真实会议场景。
- 包含摘要、问答对提取、问答三任务，评估更全面。
- 实测顶尖大模型仍有明显短板，验证了数据集挑战性。
大型语言模型（LLMs）在金融领域的突破催生了新基准评测体系。然而，现有金融基准多基于新闻、财报或公告，难以捕捉金融会议的真实动态。为此，我们提出新型基准 $ exttt{M$^3$FinMeeting}$，一个支持英语、中文和日语的多语言、多行业、多任务金融会议理解数据集。其涵盖全球行业分类标准（GICS）定义的多个行业领域，实现跨行业覆盖。数据集包含三项任务：会议摘要生成、问答对提取与问答回答，以更真实地评估模型理解能力。在七种主流大模型上的实验表明，即使最先进的长文本模型仍存在显著提升空间，证明了 $ exttt{M$^3$FinMeeting}$ 在评测金融会议理解能力方面的有效性。
原文摘要 · Abstract (English)
Recent breakthroughs in large language models (LLMs) have led to the development of new benchmarks for evaluating their performance in the financial domain. However, current financial benchmarks often rely on news articles, earnings reports, or announcements, making it challenging to capture the real-world dynamics of financial meetings. To address this gap, we propose a novel benchmark called $\texttt{M$^3$FinMeeting}$, which is a multilingual, multi-sector, and multi-task dataset designed for financial meeting understanding. First, $\texttt{M$^3$FinMeeting}$ supports English, Chinese, and Japanese, enhancing comprehension of financial discussions in diverse linguistic contexts. Second, it encompasses various industry sectors defined by the Global Industry Classification Standard (GICS), ensuring that the benchmark spans a broad range of financial activities. Finally, $\texttt{M$^3$FinMeeting}$ includes three tasks: summarization, question-answer (QA) pair extraction, and question answering, facilitating a more realistic and comprehensive evaluation of understanding. Experimental results with seven popular LLMs reveal that even the most advanced long-context models have significant room for improvement, demonstrating the effectiveness of $\texttt{M$^3$FinMeeting}$ as a benchmark for assessing LLMs' financial meeting comprehension skills.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。