用解码器做回归，效果不输传统方法还更灵活。
Decoding-based Regression
- 用解码器生成数字字符串实现回归任务
- 在标准回归任务上表现媲美传统点式头
- 适合需要平滑数值分布的任务，如密度估计
语言模型最近被证明可执行回归任务，即通过解码字符串来表示数值预测。本文从理论上论证了该能力的合理性，并进一步研究了在任意特征表示下，基于因果序列解码的模型作为数值回归头的实用性。我们发现，尽管这些解码器头以常规方式训练（通过交叉熵损失进行下一个词预测），但在标准回归任务上的表现与传统点式回归头相当，同时具备捕捉平滑数值分布的能力，例如在密度估计任务中表现优异。
原文摘要 · Abstract (English)
Language models have recently been shown capable of performing regression wherein numeric predictions are represented as decoded strings. In this work, we provide theoretical grounds for this capability and furthermore investigate the utility of causal sequence decoding models as numeric regression heads given any feature representation. We find that, despite being trained in the usual way - for next-token prediction via cross-entropy loss - decoder-based heads are as performant as standard pointwise heads when benchmarked over standard regression tasks, while being flexible enough to capture smooth numeric distributions, such as in the task of density estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。