用SOAK方法评估数据子集间模式相似性，判断能否跨域训练预测。
SOAK: Same/Other/All K-fold cross-validation for estimating similarity of patterns in data subsets
- 通过交叉验证比较不同子集训练模型在固定测试集上的表现
- 在6个真实数据集上验证了跨时空区域预测的准确性
- 适合需要评估数据异质性或合并训练集的研究者
在许多实际机器学习应用中，我们关心能否基于已有数据训练模型，并在新数据子集（如不同时期、不同地理区域）上获得准确预测。另一个问题是：各数据子集是否足够相似，值得合并训练？本文提出SOAK（Same/Other/All K-fold cross-validation），一种新方法，可同时回答上述问题。SOAK系统性地比较在不同数据子集上训练的模型在固定测试子集上的表现，以估计子集中可学习/可预测模式的相似性。我们在六个新的真实数据集（含地理/时间子集，用于检验跨子集预测精度）上进行了实验；三个图像对数据集（子集为不同类型图像，检验相似图像是否产生更小预测误差）；以及11个基准数据集（含预定义训练/测试划分，用于检验预设划分间的相似性）。结果表明，该方法能有效揭示数据子集间的模式相似性。
原文摘要 · Abstract (English)
In many real-world applications of machine learning, we are interested to know if it is possible to train on the data that we have gathered so far, and obtain accurate predictions on a new test data subset that is qualitatively different in some respect (time period, geographic region, etc). Another question is whether data subsets are similar enough so that it is beneficial to combine subsets during model training. We propose SOAK, Same/Other/All K-fold cross-validation, a new method which can be used to answer both questions. SOAK systematically compares models which are trained on different subsets of data, and then used for prediction on a fixed test subset, to estimate the similarity of learnable/predictable patterns in data subsets. We show results of using SOAK on six new real data sets (with geographic/temporal subsets, to check if predictions are accurate on new subsets), 3 image pair data sets (subsets are different image types, to check that we get smaller prediction error on similar images), and 11 benchmark data sets with predefined train/test splits (to check similarity of predefined splits).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。