arXiv:2505.16635cs.DBcs.LG2025-05

构建真实数据库图谱,评估联合学习在数据孤岛中的实际效果

WikiDBGraph: A Data Management Benchmark Suite for Collaborative Learning over Database Silos

  • 基于10万真实数据库构建带权重边的图谱,模拟复杂数据关联
  • 实测现有联合学习方法在非对齐、难连接数据上性能显著下降
  • 适合研究数据管理与联合学习融合的学者和工业界系统设计者

关系型数据库常分散于不同组织,形成阻碍分布式数据管理与挖掘的数据孤岛。联合学习(CL)通过多方协作训练模型而无需共享原始数据,为该问题提供可行方案。然而现有框架(如联邦学习、分割学习)在真实场景中部署受限:当前基准和算法多假设数据库孤立、对齐且可连接,忽略端到端数据管理流程,尤其是表连接与数据对齐等预处理环节。我们对真实世界语料库WikiDBs的分析发现,数据库间存在互联性、非对齐性和部分不可连接性,暴露了算法设计与实际部署间的巨大差距。为此,我们构建了WikiDBGraph,一个由10万真实数据库通过1700万条加权边连接而成的大规模数据集。每个节点(数据库)和边(关系)分别标注13个和12个属性,捕捉跨数据库的实例级与特征级重叠。在WikiDBGraph上的实验表明,现有联合学习方法在现实条件下既有效也存在明显局限,揭示了数据孤岛管理中此前被忽视的关键挑战,并指明了实用化联合学习系统的改进方向。

原文摘要 · Abstract (English)

Relational databases are often fragmented across organizations, creating data silos that hinder distributed data management and mining. Collaborative learning (CL) -- techniques that enable multiple parties to train models jointly without sharing raw data -- offers a principled approach to this challenge. However, existing CL frameworks (e.g., federated and split learning) remain limited in real-world deployments. Current CL benchmarks and algorithms primarily target the learning step under assumptions of isolated, aligned, and joinable databases, and they typically neglect the end-to-end data management pipeline, especially preprocessing steps such as table joins and data alignment. In contrast, our analysis of the real-world corpus WikiDBs shows that databases are interconnected, unaligned, and sometimes unjoinable, exposing a significant gap between CL algorithm design and practical deployment. To close this evaluation gap, we build WikiDBGraph, a large-scale dataset constructed from 100{,}000 real-world relational databases linked by 17 million weighted edges. Each node (database) and edge (relationship) is annotated with 13 and 12 properties, respectively, capturing a hybrid of instance- and feature-level overlap across databases. Experiments on WikiDBGraph demonstrate both the effectiveness and limitations of existing CL methods under realistic conditions, highlighting previously overlooked gaps in managing real-world data silos and pointing to concrete directions for practical deployment of collaborative learning systems.

联合学习数据孤岛数据库管理评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。