用轻量级头直接从大模型中预测数值，精度更高且参数开销极小。
RELISH: LLM REgression with a Latent Iterative State Head
- 通过迭代更新隐状态，从冻结大模型中直接回归出数值
- 在6个数据集上优于三种主流方法，误差更低
- 仅需340万参数，比同类方法少90%以上
我们提出RELISH（REgression with a Latent Iterative State Head），一种面向文本回归的新型轻量级架构。不同于将数值目标解码为文本或聚合多个生成结果，RELISH通过在标记级表示上进行交叉注意力，迭代优化一个学习到的隐状态，并使用线性回归器将最终状态映射为点估计值，直接从冻结的大语言模型表示中预测标量值。在六个数据集、四种大模型主干和两种训练范式下，RELISH始终优于三大类主流方法：自回归解码、回归感知推理及现有预测头方法。尽管性能显著提升，其参数效率极高，仅需约340万~370万可训练参数（对冻结的LLM骨干仅增加0.01%-0.04%），远低于随模型规模增长的LoRA方案（0.26%-0.42%）。代码已开源：https://github.com/SamSoup/RELISH。
原文摘要 · Abstract (English)
We present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large language models. Rather than decoding numeric targets as text or aggregating multiple generated outputs, RELISH predicts scalar values directly from frozen LLM representations by iteratively refining a learned latent state through cross-attention over token-level representations, and then mapping the final state to a point estimate with a linear regressor. Across six datasets, four LLM backbones, and two LLM training regimes, RELISH consistently outperforms prior baselines from all three major LLM regression families, including autoregressive decoding, regression-aware inference, and existing predictive head methods. Despite these gains, RELISH remains highly parameter-efficient, requiring only $\sim$3.4-3.7M trainable parameters across frozen LLM backbones (only 0.01-0.04$\%$ additional overhead), far less than LoRA-based alternatives that grow with model size (0.26-0.42$\%$). Our code is available at https://github.com/SamSoup/RELISH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。