用线性降维提升文本嵌入在小样本表格分类中的表现
Linear Dimensionality Reduction for Word Embeddings in Tabular Data Classification
- 将词嵌入分块后分别做LDA，降低协方差矩阵规模
- 带收缩的分块LDA在仅2维下仍显著优于原始嵌入
- 适合训练数据少、特征维度高的表格分类任务
工程师薪资预测挑战赛要求基于表格数据将薪资分为三类。职位描述以300维词嵌入形式加入表格特征，极大增加维度，且训练样本有限，分类难度高。针对表格数据中词嵌入的线性降维研究不足。本文研究主成分分析（PCA）与线性判别分析（LDA）。结果表明，适当选择子空间维度时，PCA可超越原始嵌入。未加正则化的LDA因协方差估计误差表现差，但引入收缩后性能显著提升，即使仅保留2维亦然。提出分块LDA（Partitioned-LDA），将嵌入均分为块，对每块独立进行LDA，有效减小协方差矩阵规模。分块LDA优于普通LDA，结合收缩后在竞赛公开排行榜达到前10名。该方法在小样本条件下有效提升词嵌入在表格分类中的性能。
原文摘要 · Abstract (English)
The Engineers' Salary Prediction Challenge requires classifying salary categories into three classes based on tabular data. The job description is represented as a 300-dimensional word embedding incorporated into the tabular features, drastically increasing dimensionality. Additionally, the limited number of training samples makes classification challenging. Linear dimensionality reduction of word embeddings for tabular data classification remains underexplored. This paper studies Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). We show that PCA, with an appropriate subspace dimension, can outperform raw embeddings. LDA without regularization performs poorly due to covariance estimation errors, but applying shrinkage improves performance significantly, even with only two dimensions. We propose Partitioned-LDA, which splits embeddings into equal-sized blocks and performs LDA separately on each, thereby reducing the size of the covariance matrices. Partitioned-LDA outperforms regular LDA and, combined with shrinkage, achieves top-10 accuracy on the competition public leaderboard. This method effectively enhances word embedding performance in tabular data classification with limited training samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。