跨异构特征空间迁移黑箱模型知识,提升新数据预测精度
Black-Box Knowledge Transfer across Distinct Feature Sets
- 分解目标函数为可迁移与不可迁移两部分,用无标签数据桥接特征空间
- 在非迁移部分小或平滑时,预测风险优于纯标签数据训练方法
- 支持多黑箱聚合,适合特征不匹配但需复用预训练模型的场景
预训练的黑箱预测函数蕴含了从海量数据和大量计算中提炼的知识。然而,当可用输入特征与黑箱期望不符时,直接使用不可行。本文提出一种将黑箱预测知识迁移到新异构输入空间的方法。该方法将目标回归函数分解为可迁移成分(黑箱可提供信息)和不可迁移成分(反映新空间特有信息)。采用两步神经网络流程:利用丰富的无标签特征对估计可迁移成分,用少量标签数据估计不可迁移成分。推导出预测风险界,在不可迁移成分小或平滑时,优于非迁移替代方案,且方法能自适应两种情况。在额外条件下,估计器最坏情形风险的多项式阶严格低于仅依赖标签数据的极小极大风险。还将框架扩展至多个黑箱,每个位于独立输入空间,证明聚合可降低预测误差,优于表现最好的单一黑箱。模拟与真实数据验证了方法的实际价值。
原文摘要 · Abstract (English)
Pre-trained black-box predictive functions encode knowledge distilled from massive datasets and extensive computation. However, when the available input features differ from those the black box expects, direct use is infeasible. We introduce a method for transferring predictive knowledge from the black box to a new, heterogeneous input space. Our approach decomposes the target regression function into a transferable component, which the black box can inform, and a non-transferable component, which captures information unique to the new space. We propose a two-step neural network procedure, estimating the transferable component from abundant unlabeled feature pairs that bridge the two input spaces and the non-transferable component from limited labels. We derive prediction risk bounds that improve on those of a non-transfer alternative when the non-transferable component is small or smooth, and the procedure adapts to either case. Under additional conditions, the worst-case risk of our estimator is of strictly smaller polynomial order than the minimax risk of estimation from the labeled data alone. We extend the framework to multiple black boxes, each on its own input space, and show that aggregation can reduce prediction error relative to the best single black box. Simulated and real data demonstrate the practical value of the method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。