让模型理解数字的单位和语义，提升复杂数值数据的推理能力。
CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics
- 设计混合编码器，将数字、范围、高斯分布与单位属性融合嵌入。
- 在DROP数据集上达到87.28%的F1，比最优基线提升9.37%。
- 适合需要精准数值理解的医疗、金融等复杂领域应用。
大型预训练模型和大语言模型在捕捉语言语义和上下文关系方面表现优异，但在处理涉及数值的任务时仍面临挑战。简单地将数值或结构化数据当作普通词汇处理是不充分的，其语义必须被模型准确理解并编码。本文提出CONE，一种混合Transformer编码器预训练模型，可将数字、范围和高斯分布嵌入到保持距离特性的向量空间中。我们设计了一种新颖的复合嵌入构造算法，将数值、范围或高斯分布与其关联的单位和属性名称整合，以精确捕捉其复杂语义。我们在跨多个领域（网络、医疗、金融、政府）的大规模数据集上进行了广泛实验，验证了CONE在数值推理方面的强大能力，在DROP数据集上取得87.28%的F1分数，相比现有最先进模型最高提升9.37%，且在Recall@10上显著优于主流模型，提升幅度高达25%。
原文摘要 · Abstract (English)
Large pre-trained models (LMs) and Large Language Models (LLMs) are typically effective at capturing language semantics and contextual relationships. However, these models encounter challenges in maintaining optimal performance on tasks involving numbers. Blindly treating numerical or structured data as terms is inadequate -- their semantics must be well understood and encoded by the models. In this paper, we propose CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance. We introduce a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics. We conduct extensive experimental evaluation on large-scale datasets across diverse domains (web, medical, finance, and government) that justifies CONE's strong numerical reasoning capabilities, achieving an F1 score of 87.28% on DROP, a remarkable improvement of up to 9.37% in F1 over state-of-the-art (SOTA) baselines, and outperforming major SOTA models with a significant Recall@10 gain of up to 25%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。