无需标注数据,自动优化大模型技能
Self-Supervised Skill Optimization

- 通过对比未标注任务实例中的行为优劣来优化技能
- 在封闭式任务上接近甚至超过依赖真实反馈的优化器
- 适合缺乏标签或奖励信号的智能体技能迭代场景
代理技能为冻结的大语言模型代理提供可复用的过程指导。近期研究显示,这类技能可通过真实反馈(GT)进行优化。然而,许多应用场景缺乏真实标签、任务评分、奖励或可靠的任务特定评估器。为此,我们提出自监督技能优化(SSO),一种仅基于未标注任务实例即可学习可复用技能的对比框架。每一步中,SSO 在未标注批次上运行当前技能,利用部分执行结果生成完整技能探针,并在同一批次上运行这些探针。一个大语言模型评判器比较结果答案、轨迹、产物或终端状态。一个独立的行为提取器识别行为差异,不依赖评判决策。SSO 利用评判结果聚合跨实例的行为支持与反对证据,按证据强度对行为排序,并从最高排名行为中重构新技能。仅当新技能在未标注验证集上表现更优时,更新才被接受。SSO 在封闭式和开放式任务上均优于现有无真实反馈的提示优化器;在封闭式基准测试中,其性能接近甚至超过最强的基于真实反馈的技能优化器,且全程未使用任何真实反馈。
原文摘要 · Abstract (English)
Agent skills provide frozen large language model (LLM) agents with reusable procedural guidance, and recent work shows that such skills can be optimized with ground-truth (GT) feedback. Many applications, however, lack GT labels, task scores, rewards, or reliable task-specific evaluators. We therefore introduce Self-Supervised Skill Optimization (SSO), a comparative framework that learns a reusable skill from unlabeled task instances alone. At each step, SSO runs the current skill on an unlabeled batch, uses a subset of the resulting executions to generate complete skill probes, and runs the probes on the same batch. An LLM judge compares the resulting answers, trajectories, artifacts, or terminal states. A separate behavior extractor identifies behavioral differences without seeing the judge's decisions. SSO uses these decisions to aggregate evidence for and against the observed behaviors across instances. It then ranks the behaviors by the resulting evidence and renders a new complete skill from the highest-ranked behaviors. The update is accepted only if the new skill outperforms the current one on an unlabeled validation set. SSO outperforms existing GT-free prompt optimizers on both closed-ended and open-ended tasks. On closed-ended benchmarks, it approaches and sometimes exceeds the strongest GT-based skill optimizer without using any GT feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。