用数学距离优化大模型数值预测，提升准确率。
Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment
- 引入平滑MMD损失函数，利用数值间距离构建核函数。
- 在4个任务上均优于交叉熵和现有数值损失方法。
- 适合需要高精度数值输出的研究与应用。
尽管大语言模型具备强大泛化能力，但在要求精确数值输出时仍不可靠。根本原因在于训练目标：标准交叉熵将数值标记视为无结构类别，忽略了其数值间的度量关系。本文提出平滑最大均值差异（SMMD），在经典MMD基础上，结合数值标记间的值距核函数与基于图的平滑性设计。通过在数值子词汇上定义核函数，SMMD通过核匹配对齐预测与目标数值分布，并在诱导的核图上平滑预测-目标残差，以促进局部一致性。我们在四个数值目标任务上评估：数学推理、算术计算、钟表时间识别和图表问答，覆盖多个开源大模型与视觉语言模型骨干网络。SMMD在各项任务中均一致优于交叉熵及近期数值目标损失；分析表明MMD与平滑性存在互补效应，凸显基于距离的核设计的重要性。代码已开源：https://github.com/Zuozhuo/smmd-loss。
原文摘要 · Abstract (English)
Despite their strong general capabilities, large language models (LLMs) often remain unreliable when outputs must be numerically precise. A key reason is the training objective: standard cross-entropy treats numeric tokens as unstructured categories and ignores the metric structure of their values. We address this mismatch with Smooth Maximum Mean Discrepancy (SMMD), which builds on the classic MMD by incorporating value-distance kernels over numeric tokens and graph-based smoothness. With this kernel defined over a numeric sub-vocabulary, SMMD aligns the predicted numeric distribution to the target via kernel matching and smooths the prediction-target residual over the induced kernel graph to encourage local consistency. We evaluate SMMD on four numeric-target tasks: mathematical reasoning, arithmetic calculation, clock-time recognition, and chart question answering, across multiple open-weight LLM and VLM backbones. SMMD consistently improves accuracy over both cross-entropy and recent numeric-target losses; analyses show complementary effects between MMD and smoothness and underscore the importance of distance-based kernel design. Code is available at https://github.com/Zuozhuo/smmd-loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。