arXiv:2512.02444cs.DBcs.LG2025-12

用强化学习发现数据表间可连接的转换关系,提升异构数据集成效率。

QJoin: Transformation-aware Joinable Data Discovery Using Reinforcement Learning

  • 通过强化学习自动学习并复用跨任务的数据转换策略。
  • 在AutoJoin基准上平均F1达91.0%,在真实数据集上提速最高7.4%。
  • 适合处理格式不一、需复杂转换的开放数据集成场景。

在大规模异构数据仓库中,发现哪些表可连接及通过何种转换连接,是数据集成与发现的核心挑战。传统等值连接方法仅适用于标识符精确匹配的情况,难以应对开放或联邦环境中不一致的格式、嵌套或跨列分割的标识符。近似连接虽能处理轻微字符串差异,但无法捕捉系统性转换。本文提出QJoin,一个基于强化学习的框架,能够学习并复用跨连接任务的转换策略。QJoin在唯一性感知奖励下训练智能体,平衡键值相似性与唯一性,探索出简洁且高价值的转换链。为加速新连接,引入两种复用机制:(i) 智能体迁移,用预训练模型初始化新策略;(ii) 转换复用,缓存成功操作序列以供相似列簇使用。在AutoJoin Web基准(31对表)上,QJoin平均F1达91.0%;在纽约+芝加哥公开数据集的19,990个连接任务中,通过复用将运行时间减少最多13,747秒(即7.4%)。结果表明,转换学习与复用可显著提升连接发现的准确率与效率。

原文摘要 · Abstract (English)

Discovering which tables in large, heterogeneous repositories can be joined and by what transformations is a central challenge in data integration and data discovery. Traditional join discovery methods are largely designed for equi-joins, which assume that join keys match exactly or nearly so. These techniques, while efficient in clean, well-normalized databases, fail in open or federated settings where identifiers are inconsistently formatted, embedded, or split across multiple columns. Approximate or fuzzy joins alleviate minor string variations but cannot capture systematic transformations. We introduce QJoin, a reinforcement-learning framework that learns and reuses transformation strategies across join tasks. QJoin trains an agent under a uniqueness-aware reward that balances similarity with key distinctiveness, enabling it to explore concise, high-value transformation chains. To accelerate new joins, we introduce two reuse mechanisms: (i) agent transfer, which initializes new policies from pretrained agents, and (ii) transformation reuse, which caches successful operator sequences for similar column clusters. On the AutoJoin Web benchmark (31 table pairs), QJoin achieves an average F1-score of 91.0%. For 19,990 join tasks in NYC+Chicago open datasets, Qjoin reduces runtime by up to 7.4% (13,747 s) by using reusing. These results demonstrate that transformation learning and reuse can make join discovery both more accurate and more efficient.

数据集成强化学习连接发现转换学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。