通过只检索最近邻样本，大幅提升表格上下文学习的推理速度。
Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN
- 用k近邻只选相似度高的训练样本，降低计算开销。
- 保留全量模型98.64%的准确率，批量推理提速2.18倍。
- 适合大规模表格数据快速推理，无需修改模型结构。
近年来，表格数据的基础模型取得了显著进展，其中TabICLv2在多个表格分类任务上达到当前最优性能。然而，全量上下文表格 ICL 仍面临随训练上下文规模增长而带来的注意力成本问题，限制了其在大数据集上的高效处理能力。局部化 TabICLv2 提出一种新方法：对每个测试样本，仅在模型第二阶段的行表示空间中检索最相似的 k 个训练邻居，而非使用全部训练上下文。该方法无需架构改动，且通过额外的第二阶段和第三阶段微调可进一步提升准确率保持水平。在 TabArena 分类任务上，经过微调的局部化模型保留了全量 TabICLv2 98.64% 的准确率，在批量推理中实现 2.18 倍中位数加速，在单查询服务场景下达到约 249 倍中位数加速。
原文摘要 · Abstract (English)
Foundational models for tabular data have made significant progress in recent years, with TabICLv2 reporting state-of-the-art performance on several tabular classification tasks. However, full-context tabular ICL still suffers from attention cost that grows with the training-context size, which limits its ability to handle large datasets efficiently. Localized TabICLv2 introduces a method that reduces the inference cost of TabICLv2 by retrieving only the k nearest training neighbours for each test point, measured by similarity in the model's Stage 2 row-representation space, rather than using the full training context. This requires no architectural changes, and we show that accuracy retention can be improved through additional Stage 2 and Stage 3 fine-tuning. On TabArena classification tasks, the fine-tuned localized model retains 98.64% of Full TabICLv2 accuracy and it achieves a median 2.18$\times$ speedup in batch inference, and reaches approximately 249$\times$ median speedup in the single-query serving setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。