用大模型嵌入增强表格数据，提升分类性能。
Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers
- 将RoBERTa和GPT-2的嵌入特征融入表格数据
- 在小样本和不平衡数据上显著提升XGBoost与CatBoost性能
- 适合处理特征少、样本少的表格分类任务
特征工程对表格数据分类任务的模型性能至关重要。本文提出一种系统方法，利用大语言模型（RoBERTa和GPT-2）嵌入来丰富表格数据特征。在多个数据集上的消融实验表明，将嵌入特征与传统数值型和类别型特征结合，可显著提升随机森林、XGBoost和CatBoost等集成分类器的预测性能，尤其在样本少、类别不平衡的数据集（如UCI Adult、Heart Disease、Titanic、Pima Indian Diabetes）上表现更优。特征重要性分析显示，LLM生成的特征常位居关键位置。本研究为基于嵌入的特征增强提供了结构化方案，验证了其在表格数据集成学习中的有效性。
原文摘要 · Abstract (English)
Feature engineering is crucial for optimizing machine learning model performance, particularly in tabular data classification tasks. Leveraging advancements in natural language processing, this study presents a systematic approach to enrich tabular datasets with features derived from large language model embeddings. Through a comprehensive ablation study on diverse datasets, we assess the impact of RoBERTa and GPT-2 embeddings on ensemble classifiers, including Random Forest, XGBoost, and CatBoost. Results indicate that integrating embeddings with traditional numerical and categorical features often enhances predictive performance, especially on datasets with class imbalance or limited features and samples, such as UCI Adult, Heart Disease, Titanic, and Pima Indian Diabetes, with improvements particularly notable in XGBoost and CatBoost classifiers. Additionally, feature importance analysis reveals that LLM-derived features frequently rank among the most impactful for the predictions. This study provides a structured approach to embedding-based feature enrichment and illustrates its benefits in ensemble learning for tabular data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。