让大模型用单个词元高效处理数字，提升计算效率
Efficient numeracy in language models through single-token number embeddings
- 用浮点数二进制表示法将任意数字编码为单个词元
- 小模型在使用该方法后几乎完美解决基础算术运算
- 适合需要高效数值计算的科学工程类应用
为推动科学与工程进展，大语言模型需高效处理大量数值数据并完成复杂计算。当前主要依赖外部工具或长推理链，导致数值表征弱化或问题长度受限。我们发现前沿大模型解决简单计算需消耗大量推理词元，这因分词策略将单个数字拆分为多个词元而加剧。因此亟需高效且有效的单词元数字编码。我们提出一组编码设计要求，并证明现有方法均未满足。为此，我们提出BitTokens：基于IEEE 754浮点数二进制表示，将任意数字映射为单个词元。通过广泛实验表明，该方法使小型语言模型几乎完美掌握基础算术算法。此效率提升可显著扩展模型能解决的问题长度与复杂度。
原文摘要 · Abstract (English)
To drive progress in science and engineering, large language models (LLMs) must be able to process large amounts of numerical data and solve long calculations efficiently. This is currently only possible through the use of external tools or extensive reasoning chains, either weakening the numerical representations of LLMs or limiting the length of problems they can solve. We show that frontier LLMs require excessive amounts of reasoning tokens to solve even basic calculations, which is exacerbated by their tokenization strategies that split single numbers into multiple tokens. This motivates the need for efficient and effective single-token number encodings. We introduce a set of desiderata for such encodings and show that existing approaches fail to fulfill them. To address these shortcomings, we propose BitTokens, a novel encoding strategy that represents any number as a single token using its IEEE 754 binary floating-point representation. Through extensive experiments we show that our BitTokens allow even small language models to learn algorithms that solve basic arithmetic operations nearly perfectly. This newly gained efficiency could expand the length and complexity of problems language models can solve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。