研究小表格数据中上下文采样对模型性能的影响,发现随机采样已足够有效。
Understanding Context Sampling in TabPFN on Small Tabular Datasets
- 通过重复采样实验验证上下文大小与采样方法对结果的影响
- 更大上下文更稳定准确,多样性比分布匹配更重要
- 随机采样效果接近复杂方法,但成本低得多,适合实用
TabPFN 通过上下文学习进行分类:它基于一组带标签的训练样本(即上下文或原型)预测测试标签,无需梯度更新。在小表格数据集上,实践者仍需选择上下文大小及具体样本。本文在15个OpenML数据集上通过重复上下文采样,研究了这些选择对预测稳定性、准确性和选择成本的影响。具体包括:(i) 更大上下文是否降低随机抽样下的预测变异性;(ii) 准确性是否依赖于训练分布的保持或特征空间覆盖;(iii) K-Means和最远点采样等昂贵方法是否优于均匀随机采样。结果显示,更大的上下文显著提升准确性和稳定性,AUC变异系数从k=16时的6%-18%降至更大规模时的1%-4%。尽管随机上下文的准确性与分布代表性相关,但控制实验表明仅匹配特征均值可导致最高0.5 AUC下降,因减少了上下文多样性。混合效应分析显示,多样性与覆盖度是准确性的更强预测因子(多样性β=+0.23,p=3×10⁻¹²;特征均值偏移β=-0.01,p=0.71)。K-Means与最远点采样虽达到类似准确率,但选择成本高出两到三个数量级。结果表明,随机采样之所以有效,是因为其在期望上提供特征空间覆盖,而非复现底层数据分布。
原文摘要 · Abstract (English)
TabPFN performs classification through in-context learning: it conditions on a set of labeled training rows (the context, or prototypes) and predicts test labels without gradient updates. On small tabular datasets, practitioners must still choose the context size and which rows constitute the context. We study how these choices affect prediction stability, accuracy, and selection cost using repeated context sampling on 15 OpenML datasets. Specifically, we investigate (i) whether larger contexts reduce prediction variability across random draws, (ii) whether accuracy depends on preserving the training distribution or on feature-space coverage, and (iii) whether expensive selection methods such as K-Means and farthest-point sampling provide benefits over uniform random sampling. We find that larger contexts are both more accurate and substantially more stable, with AUC coefficient of variation decreasing from roughly 6 to 18% at k=16 to 1 to 4% at larger context sizes on datasets with room for improvement. Although accuracy correlates with distribution representativeness in random contexts, controlled experiments show that matching feature means alone can reduce accuracy by up to 0.5 AUC because it reduces context diversity. Mixed-effects analysis identifies diversity and coverage, rather than feature-mean matching, as the stronger predictor of accuracy (diversity beta=+0.23, p=3x10^-12; feature-mean shift beta=-0.01, p=0.71). K-Means and farthest-point sampling achieve similar accuracy to random selection while requiring two to three orders of magnitude more selection cost. These results show that random sampling succeeds because it provides feature-space coverage in expectation, not because it reproduces the underlying data distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。