压缩LLM文本嵌入能提升噪声任务的回归表现
When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks
- 用自编码器无监督压缩嵌入，降低过拟合风险
- 金融收益预测等噪声任务性能提升，但高因果依赖任务下降
- 适合处理含噪回归问题的研究者参考
大型语言模型在语言建模中因规模法则表现出色，但本文表明，在基于LLM的回归任务中，压缩文本表示反而可能提升性能。研究对比了三种信噪比场景下的嵌入压缩效果：金融收益预测、写作质量评估和评论评分。结果发现，通过自编码器隐层实现的最小监督式嵌入压缩，可在噪声任务（如金融收益预测）中缓解过拟合并提升性能；但在输入与目标间存在强因果关系的任务中，压缩会降低表现。这表明可解释的压缩表示（如情感）的成功可能源于其正则化效应。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model's text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder's hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。