如何设计表格模型的嵌入流水线，提升预测效果
Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Design
- 测试256种配置，发现拼接嵌入优于替换原始列
- 大模型嵌入效果更好，但榜单排名不反映实际性能
- 梯度提升树是强下游模型，适合此类任务
嵌入是将大语言模型（LLM）的世界知识融入数据驱动机器学习模型的有效方式。然而，针对表格预测任务的LLM嵌入流水线设计尚缺乏充分实证。本文系统评估了256种流水线配置，涵盖8种预处理策略、16种嵌入模型和2种下游模型。结果表明，是否引入LLM先验知识显著依赖于具体设计：拼接嵌入通常优于用嵌入替换原列；更大规模的嵌入模型表现更优，而公开排行榜名次与模型流行度无法准确预测性能；梯度提升决策树作为下游模型表现出色。研究为表格预测中的嵌入流水线构建提供了可操作的实践指导。
原文摘要 · Abstract (English)
Embeddings are a powerful way to enrich data-driven machine learning models with the world knowledge of large language models (LLMs). Yet, there is limited evidence on how to design effective LLM-based embedding pipelines for tabular prediction. In this work, we systematically benchmark 256 pipeline configurations, covering 8 preprocessing strategies, 16 embedding models, and 2 downstream models. Our results show that it strongly depends on the specific pipeline design whether incorporating the prior knowledge of LLMs improves the predictive performance. In general, concatenating embeddings tends to outperform replacing the original columns with embeddings. Larger embedding models tend to yield better results, while public leaderboard rankings and model popularity are poor performance indicators. Finally, gradient boosting decision trees tend to be strong downstream models. Our findings provide researchers and practitioners with guidance for building more effective embedding pipelines for tabular prediction tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。