提出可测量并修复大模型对话中约束失效问题的方法
Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues
- 用契约账本记录每条约束及其检查器,提前计算最终状态
- 发现约束负载越高,模型行为滞后越严重,8B模型延迟达1.8倍
- 提前编译可显著降低失效,一句话墓碑注释恢复三分之一效果
多轮对话中用户可随意撤销或添加约束,但撤销常无效:模型仍会执行已撤回的要求(有时还附带声明已撤销的评论),这种现象称为“行为反弹”或“撤销惯性”。现有方法无法量化每条约束的影响、预测其失效风险,也无法在相同资源下修复。本文仅通过模型API实现三重突破:契约账本为每条约束配执行检查器,记录撤销为墓碑,并预先编译出最终约束状态;顺序消融探测器测量每条约束的遵守情况及累积影响;修复梯子在匹配的令牌和尝试预算下运行。在HumanEval数据集( tasks{}个任务, checkers{}个验证检查器)上,8B模型在约束负载增加时,行为滞后从 elapse_delay_two{}升至 elapse_delay_eight{},而更强模型则保持稳定。在相同检查器、模型与预算下,提前编译使反弹率显著低于无账本重试基线( estore_diff{},95%置信区间 estore_diff_ci{},p< estore_diff_p{});叠加自适应梯子干预未带来可检测增益(95%置信区间排除≥ amp_excluded_gain{})。探测器能在输出前预测反弹(AUROC amp_auroc{});一句墓碑注释可恢复约三分之一编译效果,且通过了安慰剂对照。每次结果交付开销为 otal_cost_factor{},总防护计算量占 otal_hedged{},使撤销失败成为可度量、可预测、可修复的对话状态属性。
原文摘要 · Abstract (English)
Multi-turn dialogues let users revoke constraints as easily as impose them, but revocation does not reliably take effect: models keep enacting withdrawn requirements (occasionally beneath comments asserting their removal), a failure we call \emph{behavioral relapse}, or revocation inertia. No existing instrument measures this influence per clause, predicts it before delivery, or repairs it under matched budgets. \sysname{} closes the three gaps through the model API alone: a contract ledger pairs every constraint with an executable checker, records revocations as tombstones, and compiles the net constraint state ahead of time into a single specification; a sequential ablation probe measures per-clause adherence and incremental behavioral effect; a repair ladder operates under token- and attempt-matched budgets. On \dataname{} (\NTasks{} HumanEval tasks, \NClauses{} verified checkers), relapse at an 8B operating point climbs from \ScaleDelayedMTwo{} to \ScaleDelayedMEight{} as constraint load grows, while stronger models sit at floor. Under matched checkers, model, and budget, ahead-of-time compilation significantly reduces relapse against a no-ledger verifier-retry baseline (\RestoreDiff{}, 95\% CI \RestoreDiffCI{}, $p$ \RestoreDiffP{}); adaptive ladder interventions stacked on top add no detectable gain (95\% confidence excludes gains $\geq$ \LadderExcludedGain{}). The probe predicts relapse before delivery (AUROC \AurocPrimary{}); a one-sentence tombstone note recovers about a third of the compilation effect and survives a placebo control. At \CostDeliveryFactor{} delivery overhead and \CostTotalHedged{} of API compute for every result, revocation failure becomes a measurable, predictable, and repairable property of dialogue state rather than an invisible one.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。