用大模型嵌入提升表格数据在标签-变量关系变化时的适应能力
LLM Embeddings Improve Test-time Adaptation to Tabular $Y|X$-Shifts
- 将表格数据序列化后输入大模型生成嵌入，增强表征能力
- 仅用32个标注样本即可有效适配新域，鲁棒性显著提升
- 适合关注小样本迁移与数据偏移问题的研究者
对于表格数据，由于缺失变量（即混杂因素）的存在，标签与特征之间的关系（Y|X）发生改变的情况很常见。由于无法泛化到完全未知的新领域，我们研究了在仅有少量标注样本时仍易于适配目标领域的模型。重点在于构建能缓解Y|X偏移的更丰富的表格数据表示，提出通过将表格数据序列化并输入大语言模型（LLM）来编码信息。实验发现，仅使用LLM嵌入本身带来的鲁棒性提升不一致，但基于这些嵌入训练的模型可在目标域上良好微调，即使只有32个标注样本亦可实现高效适应。该结论基于包含7650对源-目标域、261,000种模型配置及22种算法的系统性评估。该现象在不同目标数据量和适配策略下均成立。代码已公开于https://github.com/namkoong-lab/LLM-Tabular-Shifts。
原文摘要 · Abstract (English)
For tabular datasets, the change in the relationship between the label and covariates ($Y|X$-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples. We focus on building more informative representations of tabular data that can mitigate $Y|X$-shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies. The code is available at https://github.com/namkoong-lab/LLM-Tabular-Shifts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。