用混合方法提升实体消歧的效率与鲁棒性,适合企业数据审计。
Transformer-Gather, Fuzzy-Reconsider: A Scalable Hybrid Framework for Entity Resolution
- 先用预训练模型生成语义向量,再通过模糊匹配筛选候选
- 处理速度更快,召回率高达0.97,且可在普通CPU运行
- 适合需要高可靠性的企业级数据清洗场景
实体消歧在企业系统中对保障数据完整性至关重要。传统方法难以应对噪声数据或语义理解,而现代方法常面临计算开销大或过度依赖并行计算的问题。本文提出一种可扩展的混合框架,兼顾可扩展性、抗噪能力与结果可靠性。利用预训练语言模型将结构化数据编码为语义嵌入向量,随后检索语义相关候选集,并通过模糊字符串匹配技术进行句法验证,以优化未标注数据的分类。该方法应用于真实世界中的实体消歧任务,实现了中央用户管理数据库与多个共享主机服务器记录之间的关联。相比其他方法,本方案在处理时间与鲁棒性方面表现优异,系统保持约0.97的高检索召回率。框架具备良好可扩展性,可在标准CPU基础设施上部署,为企业级数据完整性审计提供实用高效解决方案。
原文摘要 · Abstract (English)
Entity resolution plays a significant role in enterprise systems where data integrity must be rigorously maintained. Traditional methods often struggle with handling noisy data or semantic understanding, while modern methods suffer from computational costs or the excessive need for parallel computation. In this study, we introduce a scalable hybrid framework, which is designed to address several important problems, including scalability, noise robustness, and reliable results. We utilized a pre-trained language model to encode each structured data into corresponding semantic embedding vectors. Subsequently, after retrieving a semantically relevant subset of candidates, we apply a syntactic verification stage using fuzzy string matching techniques to refine classification on the unlabeled data. This approach was applied to a real-world entity resolution task, which exposed a linkage between a central user management database and numerous shared hosting server records. Compared to other methods, this approach exhibits an outstanding performance in terms of both processing time and robustness, making it a reliable solution for a server-side product. Crucially, this efficiency does not compromise results, as the system maintains a high retrieval recall of approximately 0.97. The scalability of the framework makes it deployable on standard CPU-based infrastructure, offering a practical and effective solution for enterprise-level data integrity auditing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。