大模型数字表征具高度一致性，可提升算术准确性。
Language Models Learn Universal Representations of Numbers and Here's Why You Should Care
- 发现不同大模型对数字的嵌入均呈规律正弦结构。
- 数字表征在多种任务中可互换使用，效果相近。
- 增强正弦特性能有效减少模型算术错误。
先前研究表明，大型语言模型（LLMs）通常会收敛到基于正弦表示的精确数字输入嵌入。本文量化表明，这些表征实际上极为系统化，几乎完全通用：不同家族的LLM均发展出等价的正弦结构，且数字表征在大量实验设置中可广泛互换。我们证明，正确考虑这一特性对于评估LLMs编码数值及其他序数信息的准确性至关重要；机制上增强这种正弦性还能降低模型的算术误差。
原文摘要 · Abstract (English)
Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。