用大模型嵌入做回归，高维任务表现优于传统方法。
Understanding LLM Embeddings for Regression
- 将字符串转为大模型嵌入，直接用于回归预测。
- 大模型嵌入在特征空间保持光滑性，提升高维回归效果。
- 模型大小和语言理解力不总带来性能提升，值得警惕。
随着大语言模型（LLMs）在灵活处理字符串信息方面的兴起，将其应用于回归任务成为自然选择，即通过预处理将字符串表示转换为LLM嵌入，作为下游的度量预测特征。本文首次对基于嵌入的回归进行了系统研究，发现对于高维回归任务，使用LLM嵌入作为特征的表现优于传统特征工程。这一优势部分归因于LLM嵌入在数值数据上天然保持了特征空间的Lipschitz连续性。此外，我们量化了不同模型因素的影响，尤其发现模型规模和语言理解能力并不总是能提升回归性能。
原文摘要 · Abstract (English)
With the rise of large language models (LLMs) for flexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprehensive investigations into embedding-based regression and demonstrate that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. This regression performance can be explained in part due to LLM embeddings over numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。