arXiv:2604.14054cs.LGcs.CL2026-04被引 7

用自生成的解题路径提升智能体自演进效率,无需外部数据。

$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

论文配图:$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
图 1 · 摘自论文原文
  • 利用自博弈生成解题路径作为特权信息,实现密集监督。
  • 在无标注数据下超越全监督模型,进化效率提升2-3倍。
  • 适合构建低成本、高效率的自主搜索智能体系统。

深度搜索智能体在应对复杂信息查询任务方面展现出巨大潜力,但其训练仍面临奖励稀疏、信用分配困难和标注数据有限等挑战。自博弈提供了一种降低数据依赖的可扩展路径,但传统自博弈仅通过稀疏结果奖励优化学生,导致学习效率低下。本文发现,自博弈在任务生成过程中自然产生问题构建路径(QCP),该中间产物捕捉了逆向求解过程,构成一种低成本、可规模化获取的特权信息。基于此,提出π-Play框架,结合自博弈与自蒸馏:由考官生成任务及QCP,教师利用QCP作为特权上下文,通过自蒸馏对学生产生密集监督。该设计将稀疏奖励自博弈转化为密集反馈的协同演化。大量实验表明,无需数据的π-Play优于全监督搜索智能体,且进化效率较传统自博弈提升2-3倍。代码已开源。

原文摘要 · Abstract (English)

Deep search agents have emerged as a promising paradigm for addressing complex information-seeking tasks, but their training remains challenging due to sparse rewards, weak credit assignment, and limited labeled data. Self-play offers a scalable route to reduce data dependence, but conventional self-play optimizes students only through sparse outcome rewards, leading to low learning efficiency. In this work, we observe that self-play naturally produces a question construction path (QCP) during task generation, an intermediate artifact that captures the reverse solution process. This reveals a new source of privileged information: self-play can provide high-quality privileged information for the self-distillation at low cost and at scale, without relying on human feedback or curated privileged information. Leveraging this insight, we propose Privileged Information Self-Play ($π$-Play), a novel multi-agent self-evolution framework combining self-play and self-distillation. In $π$-Play, an examiner generates tasks together with QCPs, and a teacher employs QCP as privileged context to densely supervise a student via self-distillation. This design transforms sparse-reward self-play into a dense-feedback co-evolution. Extensive experiments show that data-free $π$-Play surpasses fully supervised search agents and improves evolutionary efficiency by 2-3$\times$ over conventional self-play. Code is available at https://github.com/zhyaoch/pi-play.

自博弈自蒸馏智能体演化无监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。