让大模型通过网络自建知识树,持续提升特定领域推理能力。
WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement
- 构建可扩展的领域知识树,从网络动态获取并清洗一致数据。
- 利用可验证奖励进行对抗式自博弈,实现+9.8至+14.79的性能提升。
- 支持医学、物理等领域的定向优化,适合需要精准推理的场景。
强化学习结合可验证奖励(RLVR)为语言模型自提升提供了可行路径,但现有方法存在内生自博弈易漂移、基于语料库的方法依赖预设数据集的问题。本文提出WIST——一种基于网络的迭代自博弈树框架,直接从开放网络学习,无需预设领域语料。WIST通过增量扩展领域树,检索并清洗路径一致的网络语料,构建可控训练环境;再以挑战者-求解者模式进行自博弈,利用可验证奖励反馈学习信号,更新节点后验概率并引导后续探索。在四种基线模型上,WIST均优于基础模型,且普遍超越纯内生与语料基自进化基线,整体提升达+9.8(Qwen3-4B-Base)和+9.7(OctoThinker-8B)。其具备领域可调性,在医学领域使Qwen3-8B-Base提升+14.79,在PhyBench上使Qwen3-4B-Base提升+5.28。消融实验进一步验证了关键组件对开放网络学习稳定性的重要性。代码已开源。
原文摘要 · Abstract (English)
Recent progress in reinforcement learning with verifiable rewards (RLVR) offers a practical path to self-improvement of language models, but existing methods face a key trade-off: endogenous self-play can drift over iterations, while corpus-grounded approaches rely on curated data environments. We present \textbf{WIST}, a \textbf{W}eb-grounded \textbf{I}terative \textbf{S}elf-play \textbf{T}ree framework for domain-targeted reasoning improvement that learns directly from the open web without requiring any pre-arranged domain corpus. WIST incrementally expands a domain tree for exploration, and retrieves and cleans path-consistent web corpus to construct a controllable training environment. It then performs Challenger--Solver self-play with verifiable rewards, and feeds learnability signals back to update node posteriors and guide subsequent exploration through an adaptive curriculum. Across four backbones, WIST consistently improves over the base models and typically outperforms both purely endogenous self-evolution and corpus-grounded self-play baselines, with the Overall gains reaching \textbf{+9.8} (\textit{Qwen3-4B-Base}) and \textbf{+9.7} (\textit{OctoThinker-8B}). WIST is also domain-steerable, improving \textit{Qwen3-8B-Base} by \textbf{+14.79} in medicine and \textit{Qwen3-4B-Base} by \textbf{+5.28} on PhyBench. Ablations further confirm the importance of WIST's key components for stable open-web learning. Our Code is available at https://github.com/lfy-123/WIST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。