让表格模型更小更快，还能保持灵活适应能力。
TACTICL: Task-Aware Compression of Tabular ICL Models
- 自动剪枝层并替换为轻量适配器，融合上下文与权重学习
- 可替换85%的层，下游任务性能基本不变
- 在数据变化下仍保持上下文学习能力，适合实际部署
表格任务中基础模型表现优异，但推理成本高昂。将模型蒸馏为特定任务架构虽能减小规模和计算开销，却牺牲了上下文自适应能力。本文提出TACTICL，一种面向表格上下文学习模型的任务感知压缩框架，通过联合剪枝Transformer层并用下游任务训练的轻量适配器替代，实现上下文学习与权重学习的融合。我们在47个基准数据集上评估TACTICL，结果表明可在不显著降低性能的前提下，最多替换85%的网络层。此外，TACTICL在数据分布变化下仍保持鲁棒性，维持原有的上下文学习能力。整体而言，TACTICL通过结合任务特异性适配与结构化压缩，有效利用了表格基础模型的深度冗余。代码已开源：https://github.com/Hebog/tfm_compression
原文摘要 · Abstract (English)
The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。