改进自回归模型的数值序列预测，让数字顺序更准确。
Advancing Sequential Numerical Prediction in Autoregressive Models
- 用地球移动距离改进数字顺序建模，避免独立处理每个数字
- 在多个数据集上提升数值预测精度，尤其在长序列中表现更好
- 适合需要精确数值生成的应用，如金融、科学计算
自回归模型已成为序列生成任务的主流方法，但传统方法将数字视为独立标记并使用交叉熵损失，忽略了数值序列的内在结构。本文提出数值标记完整性损失（NTIL），从两个层面改进：（1）标记级，将地球移动距离（EMD）扩展用于保持数值间的序数关系；（2）序列级，对预测序列与真实序列的整体偏差进行惩罚。该双层机制显著提升了数值序列预测性能，并可有效集成至大语言模型（LLM）和多模态大模型（MLLM）中。大量实验表明，引入NTIL后模型在多个基准数据集上均实现显著性能提升。
原文摘要 · Abstract (English)
Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。