为表格大模型设计快速优化上下文的新方法,提升小样本预测性能。
VIP-COP: Context Optimization for Tabular Foundation Models
- 基于重要性估值的显式选择机制,筛选关键样本与特征。
- 在多组数据集上显著优于基线,尤其在噪声和增强数据下表现更优。
- 无需模型内部信息,可快速部署于各类表格大模型,且结果可解释。
表格基础模型(TFMs)在结构化数据的上下文学习中展现出强大能力,可在不进行任务特定训练的情况下直接预测新任务。然而,其性能受限于上下文长度,难以处理中等规模以上的数据,且当推理时数据量超过预训练分布时性能下降。本文提出VIP-COP,通过估计训练样本与特征对预测的重要性,实现对表格基础模型的硬性上下文优化。该方法采用在线核SHAP回归、迭代精炼、价值引导采样与多保真剪枝,实现快速优化(通常几分钟内见效),具备预算感知与任意时间特性,优于依赖梯度的软提示方法,且无需访问模型内部,兼容私有与开源模型。实验表明,其在大规模高维测试集上持续超越启发式与优化基线,包括数据增强与噪声场景,建立了测试时上下文优化的新基准。
原文摘要 · Abstract (English)
Tabular foundation models (TFMs) have emerged as a powerful paradigm for in-context learning on structured data, enabling direct prediction on new tabular tasks without task-specific training. However, their effectiveness is constrained by context length limits, restricting application to medium-scale data and degrading performance when inference-time data exceed pretraining size distributions. Our work introduces VIP-COP, estimating the Value of Importance for Prediction of training examples and features for hard Context OPtimization for TFMs. Its explicit selection mechanism suppresses noise and isolates influential data, enabling the model to also benefit from data augmentation by prioritizing high-value augmented samples and features. VIP-COP is (i) fast, boosting performance often within minutes of optimization, based on an online KernelSHAP-based regression with iterative refinement, value-guided context sampling, and multi-fidelity pruning; (ii) budget-aware and any-time, improving with additional test-time compute unlike heuristics that produce fixed contexts; (iii) model-aware yet fully black-box, requiring no access to model internals, making it compatible with both proprietary and open-source TFMs; (iv) interpretable, identifying discrete ``Very Important Predictors'' (samples and features) that maximize signal-to-noise, which makes it (v) robust, isolating high-value data from noise. In contrast, soft-prompt optimization requires model gradients, produces abstract latent tokens, and lacks explicit signal discrimination. Extensive experiments show that VIP-COP consistently outperforms heuristic and optimized baselines across large-scale high-dimensional testbeds, including data augmentation and data-noise settings, establishing a new state of the art in test-time context refinement for TFMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。