无需共享特征,跨领域表格数据也能高效迁移学习。
Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

- 用Transformer投影权重构建通用上下文,实现无共享特征的跨域知识迁移。
- 在10组异构数据上平均排名第二,AUROC提升3.7%。
- 适合处理无共同字段的表格数据迁移任务,尤其适用于医疗、金融等场景。
与图像和文本不同,由于不同领域间特征类型、结构和语义的异质性,将迁移学习应用于表格数据颇具挑战。现有方法假设数据表之间存在共享特征以实现跨域知识迁移,这在实际中并不现实。本文提出广义上下文学习,消除了跨域共享特征的要求。通过Transformer的投影权重(key、value、query)捕捉的广义上下文提供基于规则的泛化能力,而非传统Transformer激活中学习到的领域特定上下文。源域的key投影权重与目标域的query权重交互,实现数据无关的跨域注意力迁移学习(CATTLE)。在十组异构源-目标数据集上的实验表明,CATTLE仅需单一源数据集即可学习广义上下文,在排名和统计上均优于九种最先进基线方法,包括机器学习、深度学习及使用大规模预训练模型的迁移学习方法。CATTLE平均排名为2.9,较基线方法平均提升3.7% AUROC。
原文摘要 · Abstract (English)
Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$, $value$, and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。