构建首个安全领域表格理解数据集，助力大模型评估
Secu-Table: a Comprehensive security table dataset for evaluating semantic table interpretation systems
- 基于CVE/CWE数据构建1500+表格，标注使用Wikidata与SEPSES CSKG
- 包含15000+实体，支持大模型在安全领域的表格到知识图谱匹配评测
- 开源代码与数据，适配学术界与工业界安全系统评估需求
评估基于大语言模型的语义表格理解系统（尤其是安全领域）严重依赖高质量数据集。然而，当前安全领域缺乏公开可用的表格数据集。本文提出Secu-Table数据集，包含超过1500张表格、超过15000个实体，数据源自通用漏洞披露（CVE）和通用弱点枚举（CWE），并通过Wikidata及安全事件流语义处理知识图谱（SEPSES CSKG）进行标注。相关代码已全部开源，并用于SemTab挑战赛中评估基于开源LLM的表格到知识图谱匹配能力。初步基线测试采用Falcon3-7b-instruct、Mistral-7B-Instruct两款开源模型以及闭源模型GPT-4o mini。
原文摘要 · Abstract (English)
Evaluating semantic tables interpretation (STI) systems, (particularly, those based on Large Language Models- LLMs) especially in domain-specific contexts such as the security domain, depends heavily on the dataset. However, in the security domain, tabular datasets for state-of-the-art are not publicly available. In this paper, we introduce Secu-Table dataset, composed of more than 1500 tables with more than 15k entities constructed using security data extracted from Common Vulnerabilities and Exposures (CVE) and Common Weakness Enumeration (CWE) data sources and annotated using Wikidata and the SEmantic Processing of Security Event Streams CyberSecurity Knowledge Graph (SEPSES CSKG). Along with the dataset, all the code is publicly released. This dataset is made available to the research community in the context of the SemTab challenge on Tabular to Knowledge Graph Matching. This challenge aims to evaluate the performance of several STI based on open source LLMs. Preliminary evaluation, serving as baseline, was conducted using Falcon3-7b-instruct and Mistral-7B-Instruct, two open source LLMs and GPT-4o mini one closed source LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。