用字符级自编码器自动分组非语义数据列,提升企业数据管理效率。
CAE: Character-Level Autoencoder for Non-Semantic Relational Data Grouping
- 基于字符级别编码,通过数据模式识别列相似性。
- 在5个最相近列匹配任务中达到80.95%准确率,远超词袋模型的47.62%。
- 适合处理大规模工业数据湖中的非语义字段,如IP、时间戳等。
企业关系型数据库中包含大量非语义数据——如IP地址、产品标识符、编码密钥和时间戳——这些数据难以通过传统语义分析处理。本文提出一种字符级自编码器(CAE)方法,通过检测数据模式与结构相似性,自动识别并分组语义相同的列。与传统自然语言处理模型在语义可解释性和词汇外词项上的局限不同,该方法在字符层面操作,并采用固定词典规模,实现对大规模数据湖和仓库的高效处理。CAE将非语义表列文本表示编码为高维特征嵌入,用于数据聚类。固定词典大小显著降低内存占用与训练时间。实验表明,该方法在跨关系数据集的前5名列匹配任务中达到80.95%的准确率,显著优于传统词袋模型(47.62%)。结果验证了其在识别和聚类关系数据集中相同列的有效性。本研究弥合了字符级神经架构理论进展与企业数据管理实际挑战之间的差距,为大规模非语义工业数据的模式理解与数据概览提供了自动化解决方案。
原文摘要 · Abstract (English)
Enterprise relational databases increasingly contain vast amounts of non-semantic data - IP addresses, product identifiers, encoded keys, and timestamps - that challenge traditional semantic analysis. This paper introduces a novel Character-Level Autoencoder (CAE) approach that automatically identifies and groups semantically identical columns in non-semantic relational datasets by detecting column similarities based on data patterns and structures. Unlike conventional Natural Language Processing (NLP) models that struggle with limitations in semantic interpretability and out-of-vocabulary tokens, our approach operates at the character level with fixed dictionary constraints, enabling scalable processing of large-scale data lakes and warehouses. The CAE architecture encodes text representations of non-semantic relational table columns and extracts high-dimensional feature embeddings for data grouping. By maintaining a fixed dictionary size, our method significantly reduces both memory requirements and training time, enabling efficient processing of large-scale industrial data environments. Experimental evaluation demonstrates substantial performance gains: our CAE approach achieved 80.95% accuracy in top 5 column matching tasks across relational datasets, substantially outperforming traditional NLP approaches such as Bag of Words (47.62%). These results demonstrate its effectiveness for identifying and clustering identical columns in relational datasets. This work bridges the gap between theoretical advances in character-level neural architectures and practical enterprise data management challenges, providing an automated solution for schema understanding and data profiling of non-semantic industrial datasets at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。