不靠逐个生成，直接从模型内部提取数值预测的分布信息。
Eliciting Numerical Predictive Distributions of LLMs Without Autoregression
- 用统计函数直接预测模型内部表示中的均值、中位数等
- 发现模型嵌入能有效反映预测的数值不确定性和分布特征
- 适合需要快速推理且关注预测不确定性的研究者
大型语言模型（LLMs）最近通过上下文学习成功应用于回归任务，如时间序列预测和表格预测。然而，其自回归解码过程对连续数值输出并不理想，因为获取数值目标的预测分布需多次采样，导致计算成本高、推理慢。本文研究是否可在无需显式自回归生成的情况下恢复预测分布特性。我们训练了一系列回归探测器，直接从模型内部表示中预测统计函数（如均值、中位数、分位数）以表征数值输出分布。结果表明，LLM嵌入包含丰富的摘要统计信息，包括数值不确定性。这揭示了模型内部编码数值不确定性的机制，并为无采样方式实现不确定性感知的数值预测提供了新可能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently been successfully applied to regression tasks -- such as time series forecasting and tabular prediction -- by leveraging their in-context learning abilities. However, their autoregressive decoding process may be ill-suited to continuous-valued outputs, where obtaining predictive distributions over numerical targets requires repeated sampling, leading to high computational cost and inference time. In this work, we investigate whether distributional properties of LLM predictions can be recovered without explicit autoregressive generation. To this end, we study a set of regression probes trained to predict statistical functionals (e.g., mean, median, quantiles) of the LLM's numerical output distribution directly from its internal representations. Our results suggest that LLM embeddings carry informative signals about summary statistics of their predictive distributions, including the numerical uncertainty. This investigation opens up new questions about how LLMs internally encode uncertainty in numerical tasks, and about the feasibility of lightweight alternatives to sampling-based approaches for uncertainty-aware numerical predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。