用提示学习让GPT-2解微分方程，效果媲美欧拉法。
From Text to Trajectories: GPT-2 as an ODE Solver via In-Context
- 把微分方程问题转为序列提示，用上下文学习让GPT-2求解。
- 演示数越多，精度呈指数提升，收敛性优于或等同欧拉法。
- 能泛化到分布外问题，展现强大外推能力。
上下文学习（ICL）已成为大语言模型的新范式，通过在提示中嵌入少量示例，使模型能够执行新任务。然而，其在自然语言处理任务中的高度非线性行为仍缺乏理解。本文探究大模型在ICL设置下求解常微分方程（ODE）的潜力。我们将标准的ODE问题及其解构造成序列提示，并评估GPT-2在这些任务上的表现。实验涵盖两类ODE，结果表明，GPT-2可有效学习一种元-ODE算法，在收敛性上与欧拉法相当或更优，并随示范数量增加实现指数级精度提升。此外，模型在分布外（OOD）问题上也表现出鲁棒的外推能力。这些实证发现为NLP中ICL的机制提供了新见解，并揭示其解决非线性数值问题的潜力。
原文摘要 · Abstract (English)
In-Context Learning (ICL) has emerged as a new paradigm in large language models (LLMs), enabling them to perform novel tasks by conditioning on a few examples embedded in the prompt. Yet, the highly nonlinear behavior of ICL for NLP tasks remains poorly understood. To shed light on its underlying mechanisms, this paper investigates whether LLMs can solve ordinary differential equations (ODEs) under the ICL setting. We formulate standard ODE problems and their solutions as sequential prompts and evaluate GPT-2 models on these tasks. Experiments on two types of ODEs show that GPT-2 can effectively learn a meta-ODE algorithm, with convergence behavior comparable to, or better than, the Euler method, and achieve exponential accuracy gains with increasing numbers of demonstrations. Moreover, the model generalizes to out-of-distribution (OOD) problems, demonstrating robust extrapolation capabilities. These empirical findings provide new insights into the mechanisms of ICL in NLP and its potential for solving nonlinear numerical problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。