给数字加'值标签'，让大模型算数更准
Value-Aware Numerical Representations for Transformer Language Models
- 用专属前缀标记显式编码数字大小
- 在各类算术任务中准确率全面超越基线
- 无需改造模型结构，适合现有大模型
基于Transformer的语言模型在数学推理任务上表现良好，但在基础数值理解与算术运算上仍易出错。根本原因在于数字作为符号标记处理，其嵌入向量未显式包含数值大小信息，导致系统性错误。本文提出一种值感知的数值表示方法，在标准分词输入基础上增加一个专门的前缀标记，其嵌入向量直接由数值大小决定。该机制将数值幅度信息直接注入模型输入空间，同时兼容现有分词器和仅解码器的Transformer架构。在算术任务上的评估显示，该方法在不同数值格式、任务类型和操作数长度下均优于基线模型。结果表明，显式编码数值大小是提升语言模型基本数值鲁棒性的有效且高效方式。
原文摘要 · Abstract (English)
Transformer-based language models often achieve strong results on mathematical reasoning benchmarks while remaining fragile on basic numerical understanding and arithmetic operations. A central limitation is that numbers are processed as symbolic tokens whose embeddings do not explicitly encode numerical value, leading to systematic errors. We introduce a value-aware numerical representation that augments standard tokenized inputs with a dedicated prefix token whose embedding is explicitly conditioned on the underlying numerical value. This mechanism injects magnitude information directly into the model's input space while remaining compatible with existing tokenizers and decoder-only Transformer architectures. Evaluation on arithmetic tasks shows that the proposed approach outperforms baselines across numerical formats, tasks, and operand lengths. These results indicate that explicitly encoding numerical value is an effective and efficient way to improve fundamental numerical robustness in language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。