为生成式事件论元抽取设计更可靠的评估框架，提升对大模型能力的准确衡量。
REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction
- 融合精确、宽松和大模型匹配，多维度评估论元抽取结果
- 在六大数据集上平均提升23.93的F1值，显著高于传统精确匹配
- 能识别隐含与分散论元，适合评估大模型在复杂语境下的表现
事件论元抽取旨在从文本中识别预定义事件角色的论元。现有方法采用精确匹配（EM）评估，要求预测论元与标注片段完全一致。该方法虽适用于基于跨度的模型，但对大语言模型（LLMs）不适用，因其常生成语义正确但形式多样的论元，导致EM严重低估性能。此外，EM无法捕捉隐含论元（未明确说出但可推断）和分散论元（跨文档分布）。为此，本文提出REGen——一种面向生成式事件论元抽取的可靠评估框架。REGen结合精确匹配、宽松匹配和基于大模型的匹配优势，更贴近人类判断。在六个数据集上的实验表明，相比EM，REGen平均提升23.93 F1，揭示了以往评估忽略的能力。人工验证进一步证实其有效性，与人类判断的一致性达87.67%。
原文摘要 · Abstract (English)
Event argument extraction identifies arguments for predefined event roles in text. Existing work evaluates this task with exact match (EM), where predicted arguments must align exactly with annotated spans. While suitable for span-based models, this approach falls short for large language models (LLMs), which often generate diverse yet semantically accurate arguments. EM severely underestimates performance by disregarding valid variations. Furthermore, EM evaluation fails to capture implicit arguments (unstated but inferable) and scattered arguments (distributed across a document). These limitations underscore the need for an evaluation framework that better captures models' actual performance. To bridge this gap, we introduce REGen, a Reliable Evaluation framework for Generative event argument extraction. REGen combines the strengths of exact, relaxed, and LLM-based matching to better align with human judgment. Experiments on six datasets show that REGen reveals an average performance gain of +23.93 F1 over EM, reflecting capabilities overlooked by prior evaluation. Human validation further confirms REGen's effectiveness, achieving 87.67% alignment with human assessments of argument correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。