给数字加结构标签,让大模型更好理解数值大小和位置
A Triadic Suffix Tokenization Scheme for Numerical Reasoning
- 将数字按三位分组并添加量级标记,明确表示千、百万等位数
- 支持从10^-15到10^18的33个量级,保留原始数字精度
- 可直接接入现有模型,适合需要精确计算的科学任务
标准子词分词会不一致地切分数字,导致大语言模型丢失数值的位置与小数结构,成为算术与科学推理错误的主要原因。本文提出三元后缀分词(Triadic Suffix Tokenization, TST),将数字按三位一组分组,并为每组添加显式的量级标记。该方法为整数部分(千、百万、十亿等)建立固定一对一映射,同时对小数部分(十分之一、千分之一等)采用重复标记系统。相比依赖位置推断的方法,TST提供稳定梯度信号,有利于模型收敛。提出两种实现方式:(1) 基于词汇表的方案,最多增加10,000个固定词元,覆盖10^-15至10^18共33个量级;(2) 使用少量特殊标记动态表示量级。两者均保持原始数字完整,且在词元层面清晰表达数量级关系。虽然以三位分组为基础,但框架可扩展至任意分组大小,支持线性词汇量增长以适应任意精度与范围。TST与模型架构无关,可作为即插即用的预处理步骤。实验验证留待后续工作。
原文摘要 · Abstract (English)
Standard subword tokenization methods fragment numbers inconsistently, causing large language models (LLMs) to lose positional and decimal structure - a primary driver of errors in arithmetic and scientific reasoning. We introduce Triadic Suffix Tokenization (TST), a deterministic scheme that partitions digits into three-digit triads and annotates each triad with an explicit magnitude marker. Critically, the scheme defines a fixed, one-to-one mapping between suffixes and orders of magnitude for the integer part (thousands, millions, billions, etc.) and a parallel system of replicated markers for fractional depth (tenths, thousandths, millionths, etc.). Unlike approaches that rely on positional inference, this method provides a consistent gradient signal, which should ensure stable convergence. Two implementation variants are proposed: (1) a vocabulary-based approach that adds at most 10,000 fixed tokens to an existing vocabulary, covering 33 orders of magnitude ($10^{-15}$ to $10^{18}$); and (2) a suffix-marker approach that uses a small set of special tokens to denote magnitude dynamically. Both variants preserve exact digits while making order-of-magnitude relationships transparent at the token level. While we focus on 3-digit groups (Triadic), the framework is inherently scalable to any group size for precise vocabulary optimization. Furthermore, it allows for linear vocabulary expansion to accommodate arbitrary precision and range. TST is architecture-agnostic and can be integrated as a drop-in preprocessing step. Experimental validation is deferred to future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。