让大模型当导师,自动调整指令难度以匹配智能体能力
LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

- 用大模型同时做评判和生成,动态添加约束提升指令难度
- 在三个复杂任务上性能超越静态提示和仅调评分标准的方法
- 适合研究非可验证指令遵循的强化学习与智能体训练
非可验证指令遵循的强化学习越来越依赖带有特定评分标准的大模型裁判作为奖励信号。尽管近期方法会随策略演化调整评分标准,但训练提示仍保持静态,来自固定语料库。这种静态方式常导致提示难度与策略能力严重错配,当提示无法引发不同回放间的质量差异时,裁判难以提供有区分度的奖励信号。为解决此问题,我们提出 LLM-as-a-Tutor 框架,将大模型角色从裁判扩展为导师:单一模型既作为评判者,通过成对比较策略回放检测非挑战性提示,又作为生成者,向提示中追加原子级约束。该追加式设计随策略能力单调提升难度,无需外部难度调度即可实现自校准训练信号。在三个复杂指令遵循基准测试中,本方法持续优于不感知策略的基线及先前仅适配评分标准或重写提示的方法,表明提示适应是当前非可验证强化学习中缺失的策略感知维度。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。