arXiv:2603.06333cs.AIcs.CL2026-03被引 1

提出SAHOO框架,解决自迭代系统中的对齐漂移问题。

SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement

  • 用多信号检测器监控目标漂移,融合语义与结构特征。
  • 在代码和推理任务中分别提升18.3%和16.8%性能。
  • 适合关注AI自我优化安全性的研究者与开发者。

递归自改进正从理论走向实践:现代系统可自主批判、修正与评估输出,但迭代修改可能引发隐性对齐漂移。本文提出SAHOO,一种实用框架,通过三项防护机制控制漂移:(i) 目标漂移指数(GDI),一个结合语义、词汇、结构与分布度量的可学习多信号检测器;(ii) 约束保持检查,强制执行如语法正确性与无幻觉等安全关键不变量;(iii) 回归风险量化,标记回退先前成果的改进周期。在189个任务(涵盖代码生成、数学推理与真实性)中,SAHOO实现显著质量提升,代码任务增益18.3%,推理任务增益16.8%,并在两个领域保持约束符合,在真实性上维持低违规率。阈值基于18个任务的小型验证集校准,覆盖三个迭代周期。进一步绘制能力-对齐前沿,揭示早期改进高效但后期对齐成本上升,并暴露流畅性与事实性之间的领域特异性矛盾。SAHOO因此使递归自改进过程中的对齐保持可测量、可部署、且大规模系统验证。

原文摘要 · Abstract (English)

Recursive self-improvement is moving from theory to practice: modern systems can critique, revise, and evaluate their own outputs, yet iterative self-modification risks subtle alignment drift. We introduce SAHOO, a practical framework to monitor and control drift through three safeguards: (i) the Goal Drift Index (GDI), a learned multi-signal detector combining semantic, lexical, structural, and distributional measures; (ii) constraint preservation checks that enforce safety-critical invariants such as syntactic correctness and non-hallucination; and (iii) regression-risk quantification to flag improvement cycles that undo prior gains. Across 189 tasks in code generation, mathematical reasoning, and truthfulness, SAHOO produces substantial quality gains, including 18.3 percent improvement in code tasks and 16.8 percent in reasoning, while preserving constraints in two domains and maintaining low violations in truthfulness. Thresholds are calibrated on a small validation set of 18 tasks across three cycles. We further map the capability-alignment frontier, showing efficient early improvement cycles but rising alignment costs later and exposing domain-specific tensions such as fluency versus factuality. SAHOO therefore makes alignment preservation during recursive self-improvement measurable, deployable, and systematically validated at scale.

自改进对齐保障机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。