arXiv:2606.10064cs.LGcs.AI2026-06

用激励对齐的智能体竞技场生成高质量训练轨迹,显著提升小模型购物代理性能。

Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces

  • 构建Bittensor上的购物代理竞技场,通过机制设计保证轨迹多样性与判断真实性。
  • 模型在保留数据上从18.0%准确率提升至42.7%,接近合成数据基线43.6%。
  • 释放过滤器、数据集和竞技场机制,适合研究小模型训练与智能体数据生成者。

小模型智能体后训练的瓶颈不在于算法,而在于其消耗的轨迹数据质量。现有方法(如RLVR、组相对强化学习、拒绝采样再SFT)均需多轮带轨迹监督的轨迹数据,但两个现有来源均不足:前沿合成数据继承合成器偏差并压缩长尾分布,未过滤的生产日志未经评判且含捷径行为。本文提出可工程化设计激励对齐的智能体竞技场来生成此类轨迹,并在ShoppingBench子网SN15上验证。该子网通过竞赛机制、LLM推理裁判和轮换泄漏簇防护任务集,生成具备三特性数据:激励对齐多样性、每轨迹判别性、防记忆的留出评估。引入结构质量过滤器,仅保留智能体自身发出工具调用的轨迹,剔除仅分类或叙述确定性搜索循环的子任务轨迹,进而使用与ShoppingBench公开流程匹配的SFT-then-GRPO方法对Qwen3-4B进行微调。在泄漏簇防护的留出分区、生产严格评分下,模型准确率从原始基线18.0%提升至42.7%,接近合成数据仅SFT基线43.6%水平,且仅用不到一天子网输出量。监督栈仍存在大差距(pass@8: 53.3% vs pass@1: 34.8%),通过每步教师引导的Dr. GRPO奖励将此空间转化为过程改进,识别出子任务数据流是缩小与48.7% SFT+GRPO目标差距的主要杠杆。论文开源过滤器、数据集划分及竞技场机制。

原文摘要 · Abstract (English)

Small-model agentic post-training is bottlenecked less by the algorithm than by the trajectory substrate it consumes. Leading recipes (RLVR, group-relative RL, rejection-sampled re-SFT) all need multi-turn traces carrying per-trajectory supervision, and the two existing sources fall short: frontier-synthesised data inherits the synthesizer's biases and collapses the long tail, while unfiltered production logs are unjudged and contaminated by shortcut behaviour. We argue that an incentive-aligned agent arena can be engineered to manufacture such trajectories, and demonstrate this on ORO Subnet 15 (SN15), a Bittensor deployment of the ShoppingBench agentic-commerce benchmark. SN15's race mechanism, LLM reasoning judge, and rotating leak-cluster-guarded problem suite yield a corpus with three properties: incentive-aligned diversity, per-trajectory judging, and anti-memorised held-out evaluation. We introduce a structural-quality filter that converts the raw firehose into a trainable corpus by keeping agentic trajectories (the model itself emits the tool calls) and rejecting sub-task trajectories (the model only classifies or narrates over a deterministic search loop), then post-train Qwen3-4B with a recipe matched to the published ShoppingBench SFT-then-GRPO pipeline. On a leak-cluster-guarded held-out partition scored production-strict, the model lifts from the published Qwen3-4B base of 18.0% ASR to 42.7%, within single-problem noise of the synthetic-data SFT-only baseline (43.6%), while training on a fraction of a single day of subnet output. The supervised stack leaves a large pass@8 to pass@1 gap (53.3% vs 34.8%); a per-step teacher-grounded Dr. GRPO reward converts that headroom into process improvement, and we identify the sub-task firehose as the primary lever for closing the gap to the 48.7% SFT+GRPO bar. We release the filter, the corpus splits, and the arena mechanics.

智能体训练数据生成强化学习购物代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。