用预训练语言模型解决跨学科科学问题,性能媲美专用模型。
BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask Learner
- 通过理论-实验学习框架融合文本与数值数据
- 实现50位数加法100%准确,多任务表现超越基准
- 适合科研人员和科学计算方向的开发者使用
我们提出BigBang-Proton,一种基于序列的统一架构,通过在跨尺度、跨结构、跨学科的真实科学任务上预训练,构建科学多任务学习器。相比主流通用大模型,其三大创新为:理论-实验学习范式将大规模数值实验数据与理论文本语料对齐;二进制块编码替代传统BPE分词;蒙特卡洛注意力取代传统Transformer结构。在混合了真实世界科学数据集与通用文本语料的下一步词预测预训练后,经微调与推理,BigBang-Proton在50位数加法中达100%准确率,在粒子物理喷注识别上表现与领先专用模型相当,在原子间势能模拟中达到与专用模型相当的平均绝对误差(MAE),在水质预测中表现可比传统时空模型,并在基因组建模中超越基准。结果表明,语言引导的科学计算可媲美甚至超越专用科学模型,同时保持多任务能力。我们进一步提出将预训练扩展至宇宙尺度,作为构建物质世界基础模型的关键一步。
原文摘要 · Abstract (English)
We introduce BigBang-Proton, a unified sequence-based architecture for auto-regressive language modeling pretrained on cross-scale, cross-structure, cross-discipline real-world scientific tasks to construct a scientific multi-task learner. BigBang-Proton incorporates three fundamental innovations compared to mainstream general-purpose LLMs: Theory-Experiment Learning paradigm aligns large-scale numerical experimental data with theoretical text corpora; Binary Patch Encoding replaces byte pair encoding(BPE) tokenization; Monte Carlo Attention substitutes traditional transformer architectures. Through next-word-prediction pretraining on cross-discipline scientific datasets of real-world problems mixed with general textual corpus, followed by fine-tuning and inference on downstream tasks, BigBang-Proton demonstrates 100\% accuracy in up to 50-digit arithmetic addition operations, performance on par with leading specialized models in particle physics jet tagging, matching MAE of specialized models in inter-atomic potential simulation, performance comparable to traditional spatiotemporal models in water quality prediction, and benchmark-exceeding performance in genome modeling. These results prove that language-guided scientific computing can match or exceed the performance of task-specific scientific models while maintaining multitask learning capabilities. We further hypothesize to scale the pretraining to the universe scale as a fundamental step toward developing material world foundational model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。