仅用元数据给表格列头做语义标注,挑战传统依赖数据的方法。
Column Vocabulary Association (CVA): semantic interpretation of dataless tables
- 基于元数据和零样本设置,用大模型与检索增强生成标注列头。
- 温度低于1.0时大模型准确率达100%,部分场景表现极佳。
- 组织相关数据下传统方法反而优于大模型,凸显数据关联影响。
传统语义表解析(STI)主要依赖表中实际数据进行语义标注。今年的SemTab挑战赛设立了“元数据到知识图谱”赛道,聚焦于仅使用元数据信息完成STI,而不访问底层数据。为此,本文提出新任务:列词汇关联(CVA),即仅基于元数据对列标题进行语义标注。我们评估了多种方法在该任务上的表现,包括大语言模型(LLMs)与检索增强生成(RAG)方法,以及传统的基于SemanticBERT的相似度方法。所有实验均采用零样本设定,无预训练或示例输入,避免领域特定偏差。共测试7种模型:3个商业GPT模型(gpt-3.5-turbo-0.125、gpt-4o、gpt-4-turbo)和4个开源模型(llama3-80b、llama3-7b、gemma-7b、mixtral-8x7b)。通过RAG系统集成,并探索不同温度参数对性能的影响。同时,以SemanticBERT为基础分析不同元数据类型对其表现的影响。初步结果表明,当温度低于1.0时,大模型在某些情况下准确率可达100%。然而,研究也发现数据本身的性质显著影响结果:当输入数据与术语表由同一机构创建时,传统方法反而超越大模型表现。
原文摘要 · Abstract (English)
Traditional Semantic Table Interpretation (STI) methods rely primarily on the underlying table data to create semantic annotations. This year's SemTab challenge introduced the ``Metadata to KG'' track, which focuses on performing STI by using only metadata information, without access to the underlying data. In response to this new challenge, we introduce a new term: Column Vocabulary Association (CVA). This term refers to the task of semantic annotation of column headers solely based on metadata information. In this study, we evaluate the performance of various methods in executing the CVA task, including a Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) approach, as well as a more traditional similarity approach with SemanticBERT. Our methodology uses a zero-shot setting, with no pretraining or examples passed to the Large Language Models (LLMs), as we aim to avoid a domain-specific setting. We investigate a total of 7 different LLMs, of which three commercial GPT models (i.e. gpt-3.5-turbo-0.125, gpt-4o and gpt-4-turbo) and four open source models (i.e. llama3-80b, llama3-7b, gemma-7b and mixtral-8x7b). We integrate this models with RAG systems, and we explore how variations in temperature settings affect performances. Moreover, we continue our investigation by performing the CVA task utilizing SemanticBERT, analyzing how various metadata information influence its performance. Initial findings indicate that LLMs generally perform well at temperatures below 1.0, achieving an accuracy of 100\% in certain cases. Nevertheless, our investigation also reveal that the nature of the data significantly influences CVA task outcomes. In fact, in cases where the input data and glossary are related (for example by being created by the same organizations) traditional methods appear to surpass the performance of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。