让大模型更好理解数字，提升数学解题能力
Interleaving Text and Number Embeddings to Solve Mathemathics Problems
- 用MLP为不同数字分配独立嵌入方向，增强表达力
- 在10⁻³到10⁸范围内实现R²=0.9988的高精度预测
- 设计路由层区分文本与数字嵌入，减少数值偏差
有效融合文本与数值是提升大语言模型在科学任务中表现的关键。现有方法多依赖数字离散化（如科学记数法或十进制分解），而近期研究提出连续数值编码作为归纳偏置。本文在此基础上引入更丰富的数值嵌入表示，解决数值伪影与量级截断问题。主要贡献包括：首先，使用MLP为不同数字分配嵌入空间中的独特方向；其次，引入路由层以区分数值与文本嵌入。该方法使模型能同时区分文本与数值分布并保持算术能力。仅用45M参数的编码器-解码器架构，在10⁻³至10⁸量级范围内实现R²=0.9988的预测性能，且实证显示数值伪影与偏差显著降低。
原文摘要 · Abstract (English)
Integrating text and numbers effectively is a crucial step towards enhancing Large Language Models (LLMs) capabilities in assisting in scientific tasks. While most current approaches rely on discrete tokenization of numbers, for instance, conversion to scientific notation or base 10-decomposition, a recent approach proposed a continuous numerical encoding as an inductive bias. In this paper, we build upon this approach by introducing more expressive numerical embeddings. Our method addresses key shortcomings, including the elimination of numerical artefacts and the ability to handle a wide range of magnitudes without clipping. Our work presents two key contributions. First, we employ an MLP to assign distinct directions in the embedding space to different numbers. Our second contribution is the introduction of a routing layer that differentiates between numerical and text embeddings. We hypothesise that this combined approach enables the model to distinguish between text and number distributions while maintaining its capacity for arithmetic operations. Using only a 45 M parameter encoder-decoder architecture our method achieves a $R^2$=0.9988 over a wide range of magnitude ($10^{-3},10^{8}$). In addition, we empirically observe a reduction of the numerical artefacts and biases observed compared to the baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。