让小模型在关键时刻求助大模型,降低推理成本。
R2V Agent: Teaching SLMs When to Ask for Help

- 小模型自主决策,仅在风险高时才触发大模型介入。
- 在多个任务上实现90%以上成功率,大模型调用率低于一半。
- 适合需要高效推理的交互式系统,如智能助手、自动化工具。
高效智能体应在本地小模型(SLM)可能失败时才调用昂贵的大模型(LLM)。现有方法多在执行前整体路由请求,但任务难度常在执行中变化——如工具调用失败、观察截断或错误累积,导致预判失效。本文提出R2V-Agent,一种基于风险校准的SLM-LLM路由框架。其包含:一个蒸馏的小语言模型策略、更强的教师大模型、轻量级过程验证器(每步评分候选动作)和校准的步骤级路由器。路由器为核心创新:在小模型训练后,实时估算每步残余失败风险,仅在必要时才升级至教师干预。为明确定义路由问题,先通过行为克隆(BC)与验证器引导的直接偏好优化(DPO)训练稳定小模型;再以固定策略的残差失败数据,使用布里尔校准概率估计和条件风险价值(CVaR)约束目标训练路由器,惩罚最坏情况下的失败。在HumanEval+、TextWorld和TerminalBench上测试,四种小模型骨干均表现优异:人类评估任务成功率达94.3%,大模型调用仅0.60%;TextWorld从64.6%提升至98.2%,大模型调用41.7%;TerminalBench达93.3%成功,大模型调用33.9%,约为启发式路由器成本的一半。
原文摘要 · Abstract (English)
Efficient agentic systems should incur expensive frontier-model costs only on decisions where a cheaper local model is likely to fail. Existing LLM cascades usually route whole queries before execution, but task difficulty shifts mid-trajectory - after flaky tool calls, truncated observations, or compounding local errors - making pre-execution routing brittle. We introduce \textbf{R2V-Agent}, a risk-calibrated SLM-LLM routing framework for interactive agents. R2V combines four components: a distilled small language model (SLM) policy, a stronger teacher LLM, a lightweight process verifier that scores candidate actions at each step, and a calibrated step-level router. The router is our central contribution: after the SLM is trained, it estimates residual failure risk at each step and escalates only when teacher intervention is warranted. To make the routing problem well-defined, we first train a stable local SLM using a standard offline pipeline: behavioral cloning (BC) on teacher trajectories, followed by verifier-guided Direct Preference Optimization (DPO) with consistency regularization. The router is then trained on this fixed policy's residual failures using Brier-calibrated probability estimation and a Conditional Value-at-Risk (CVaR)-constrained objective that penalizes worst-case failures across perturbation seeds. Across HumanEval+, TextWorld, and TerminalBench with four SLM backbones, R2V improves the reliability-cost frontier: it achieves $94.3\%$ HumanEval+ success with $0.60\%$ LLM escalation, recovers TextWorld from $64.6\%$ SLM-only success to $98.2\%$ at $41.7\%$ escalation, and reaches $93.3\%$ TerminalBench success at $33.9\%$ LLM calls, roughly half the heuristic-router cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。