用物理模拟器生成训练数据，让大模型学会解国际物理奥赛题。
Solving Physics Olympiad via Reinforcement Learning on Physics Simulators
- 用物理引擎生成随机场景和问答对，通过强化学习训练大模型。
- 仅用模拟数据训练，模型在国际物理奥赛上性能提升5-10个百分点。
- 适合对物理推理、仿真训练感兴趣的科研与教育从业者。
随着DeepSeek-R1等大模型推理能力的显著提升，其进展主要依赖互联网上的问答对数据，但这类数据在数学以外的科学领域（如物理）规模有限且分布不均。本文提出，物理模拟器可作为训练大模型进行物理推理的强大替代数据源。我们通过物理引擎生成随机场景，基于模拟交互自动生成合成问答对，并利用强化学习在该合成数据上训练大模型。实验表明，模型在未见过的真实物理问题上实现零样本模拟到现实的迁移：仅使用模拟数据训练，即可在国际物理奥赛（IPhO）问题上提升5至10个百分点的表现，且适用于不同模型规模。结果证明，物理模拟器能作为可扩展的数据生成器，使大模型突破互联网问答数据的局限，掌握深层次物理推理能力。代码已公开：https://sim2reason.github.io/。
原文摘要 · Abstract (English)
We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question-answer (QA) pairs, a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。