用LLM自动淘汰低效技能,确保系统不退化。
Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?
- 基于技能贡献度动态淘汰,限制库宽与判断阈值
- 实测提升测试通过率0.328个百分点,避免技能库漂移
- 适用于自进化AI系统,尤其关注可靠性评估
一个能自写和修改技能库的大型语言模型(LLM)代理,必须决定哪些技能保留,但每个技能仅有一个噪声标量评分。研究发现:若判错率(将失败误判为通过)达到或超过(1−τ)/2,无论样本量多大,都无法淘汰任何技能。审计显示,由LLM编写的技能相比无技能控制组仅提升0.0个百分点,而人类编写的则提升16.2个百分点。若不维护,技能库会进入“库漂移”状态,最终注入新技能反而比不注入更差。本文提出「Ratchet」机制,根据实际贡献淘汰技能,限制库宽为C,约束生成过程,使在难的MBPP+子集上持有样本的通过率提升0.328个百分点。该机制的非发散性边界仅在两个条件下成立:库宽C和剔除阈值τ。本工作揭示了当前部署系统中缺失的关键条件。在无参考领域中,评分来自LLM裁判,其两类错误——误判失败为通过、误判通过为失败——行为迥异。前者损失采样效率,可通过更多试验弥补;后者扭曲淘汰统计,规则内无法修正。最终任务得分作为预警信号效果差,变化幅度不超过调控提升的五分之一,且非单调。本文证明了可认证区域的两个边界,并在实时循环中验证,单次离线即可判定裁判偏向。
原文摘要 · Abstract (English)
A large language model (LLM) agent that writes and edits its own skill library must also decide which skills to keep, from one noisy scalar per skill. The answer is exact: a judge scoring failures as passes at rate $(1-τ)/2$ or above retires nothing, at any sample size, for eviction margin $τ$. Audits find that machinery is rarely built: LLM-written skills are worth $+0.0$ percentage points (pp) against a no-skill control, human-written ones $+16.2$pp. Unmaintained, a library enters \emph{library drift}, growing until injecting a skill scores worse than injecting nothing. \textbf{Ratchet} repairs this: it evicts each skill on its measured contribution, caps the library at width $C$, and constrains synthesis, lifting held-out $pass@1$ by $+0.328$ on a hard MBPP+ slice. The matching non-divergence bound is finite for exactly two reasons, $C$ and $τ$. Our contribution is the condition this repair carries and no deployed system states. In reference-free domains the scalar comes from an LLM judge, whose two error directions, modelled as a binary channel, behave nothing alike. Passes scored as failures cost sample efficiency, which more trials buy back; failures scored as passes displace the eviction statistic, and no correction inside the rule recovers it. End-task score is a poor alarm, moving by at most a fifth of the governed lift and not monotonically in the rate. We prove both edges of the certifiable region, confirm them in a running loop, and place a judge on a known side in one offline pass.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。