arXiv:2606.18812cs.LGcs.AI2026-06

用合成数据训练强化学习基础模型,实现零样本泛化。

Reinforcement Learning Foundation Models Should Already Be A Thing

论文配图:Reinforcement Learning Foundation Models Should Already Be A Thing
图 1 · 摘自论文原文
  • 用合成马尔可夫决策过程构建先验,替代真实数据收集
  • 模型在未见过的表格任务上,在线学习少于10次迭代即收敛
  • 适合追求零样本泛化能力的强化学习研究者

语言和视觉领域的基础模型依赖互联网规模数据,而结构化任务如表格预测则依赖合成数据。这一转变将挑战从数据收集转向先验设计。许多结构化任务已有现成先验:如基于合成贝叶斯先验预训练的TabPFN及其变体,能高效解决表格分类问题。本文提出两点:第一,强化学习是显著空白——生成合成马尔可夫决策过程(MDP)与生成合成表格数据同样可行,但现有上下文强化学习工作未将先验设计作为核心目标;第二,MDP具有固定大小的充分统计量,形状为表格且独立于观测轨迹,可直接适配用于表格基础模型的注意力架构,仅需将监督目标替换为策略头。这定义了强化学习基础模型的研究议程。作为概念验证,我们完全基于合成MDP训练图注意力网络,在无需任务特定调优的情况下,即可在保留的表格基准上实现上下文推理:在线学习中,所需迭代次数远少于UCB-VI和表格Q-learning;离线学习中,性能媲美VI-LCB。

原文摘要 · Abstract (English)

Foundation models for language and vision are powered by internet-scale data, while structured domains such as tabular prediction are powered by synthetic data. This substitute shifts the challenge from collection to prior design. Such priors already exist for many structured tasks: TabPFN and its successors solve tabular classification with a transformer pretrained on a synthetic Bayesian prior. We make two points. \textbf{First}, reinforcement learning is the conspicuous gap: sampling a synthetic MDP is as feasible as sampling a synthetic tabular dataset, yet no in-context RL work treats prior design as a primary objective. \textbf{Second}, MDPs admit a fixed-size sufficient statistic, independent of the episodes observed and tabular in shape, which makes them directly amenable to the attention-based architectures used for tabular foundation models, with a policy head replacing the supervised target. Together these define the agenda for an RL foundation model. As a proof of concept, we train a Graph Attention Network entirely on synthetic MDPs and show that, with no task-specific tuning, it solves held-out tabular benchmarks in context, both online and offline: online, in far fewer episodes than UCB-VI and tabular Q-learning, and offline, competitively with VI-LCB.

强化学习基础模型合成数据零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。