IntSeqBERT用模谱嵌入预测数列，显著提升大数和周期结构的建模能力。
IntSeqBERT: Learning Arithmetic Structure in OEIS via Modulo-Spectrum Embeddings
- 双流Transformer：对数尺度嵌入+100个模数的sin/cos嵌入，通过FiLM融合
- 大模型在测试集上达95.85%大小准确率、50.38%平均模数准确率，优于基线8.9和4.5个百分点
- 模数信息贡献关键性能提升，基于中国剩余定理的解码器使下一項預測提升7.4倍
OEIS中的整数序列涵盖从个位数常量到天文级阶乘与指数的数值范围，标准分词模型难以处理未见数值且无法利用周期性算术结构。本文提出IntSeqBERT，一种用于OEIS的双流Transformer编码器，对每个序列元素沿两个互补维度编码：连续对数尺度幅度嵌入，以及针对100个模数（2至101）的sin/cos模嵌入，通过FiLM融合。三个预测头（幅度回归、符号分类、100个模数的模预测）在274,705条OEIS序列上联合训练。大型模型（9150万参数）在测试集上实现95.85%的幅度准确率和50.38%的平均模数准确率（MMA），分别比标准分词Transformer基线提升8.9和4.5个百分点。消融实验表明，移除模数流导致MMA下降15.2个百分点，并额外降低幅度准确率6.2个百分点。基于概率中国剩余定理（CRT）的解码器将模型输出转化为具体整数，使下一項预测准确率相比基线提升7.4倍（Top-1：19.09% vs. 2.59%）。模谱分析揭示归一化信息增益（NIG）与欧拉函数比值φ(m)/m呈强负相关（r = -0.851, p < 10⁻²⁸），实证表明合数模能通过CRT聚合更高效地捕捉OEIS算术结构。
原文摘要 · Abstract (English)
Integer sequences in the OEIS span values from single-digit constants to astronomical factorials and exponentials, making prediction challenging for standard tokenised models that cannot handle out-of-vocabulary values or exploit periodic arithmetic structure. We present IntSeqBERT, a dual-stream Transformer encoder for masked integer-sequence modelling on OEIS. Each sequence element is encoded along two complementary axes: a continuous log-scale magnitude embedding and sin/cos modulo embeddings for 100 residues (moduli $2$--$101$), fused via FiLM. Three prediction heads (magnitude regression, sign classification, and modulo prediction for 100 moduli) are trained jointly on 274,705 OEIS sequences. At the Large scale (91.5M parameters), IntSeqBERT achieves 95.85% magnitude accuracy and 50.38% Mean Modulo Accuracy (MMA) on the test set, outperforming a standard tokenised Transformer baseline by $+8.9$ pt and $+4.5$ pt, respectively. An ablation removing the modulo stream confirms it accounts for $+15.2$ pt of the MMA gain and contributes an additional $+6.2$ pt to magnitude accuracy. A probabilistic Chinese Remainder Theorem (CRT)-based Solver converts the model's predictions into concrete integers, yielding a 7.4-fold improvement in next-term prediction over the tokenised-Transformer baseline (Top-1: 19.09% vs. 2.59%). Modulo spectrum analysis reveals a strong negative correlation between Normalised Information Gain (NIG) and Euler's totient ratio $φ(m)/m$ ($r = -0.851$, $p < 10^{-28}$), providing empirical evidence that composite moduli capture OEIS arithmetic structure more efficiently via CRT aggregation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。