arXiv:2508.17621cs.CLcs.AI2025-08NeurIPS被引 7

动态判断干预时机与强度,让大模型生成更符合预期内容。

Steering When Necessary: Flexible Steering Large Language Models with Backtracking

  • 根据问题和生成内容实时追踪模型状态,决定是否干预
  • 引入回溯机制修正错误输出,提升生成准确性
  • 在TruthfulQA等数据集上效果优于现有方法

大语言模型在诸多生成任务中表现卓越,但有效对齐其行为仍具挑战。激活值调制是一种高效低成本的方法,在推理阶段直接修改模型激活值以引导生成结果,避免微调的高昂开销。现有方法通常对所有生成无差别干预或仅依据问题决定干预,难以精准评估干预强度。为此,我们提出灵活激活调制回溯框架(FASB),通过追踪生成过程中的内部状态,结合问题与生成内容动态判断干预必要性与强度。由于发现偏差后才干预往往已太晚,我们进一步设计回溯机制,修正已生成的偏离内容,引导模型回归期望行为。在TruthfulQA及六个选择题数据集上的大量实验表明,本方法显著优于基线。代码将公开于https://github.com/gjw185/FASB。

原文摘要 · Abstract (English)

Large language models (LLMs) have achieved remarkable performance across many generation tasks. Nevertheless, effectively aligning them with desired behaviors remains a significant challenge. Activation steering is an effective and cost-efficient approach that directly modifies the activations of LLMs during the inference stage, aligning their responses with the desired behaviors and avoiding the high cost of fine-tuning. Existing methods typically indiscriminately intervene to all generations or rely solely on the question to determine intervention, which limits the accurate assessment of the intervention strength. To this end, we propose the Flexible Activation Steering with Backtracking (FASB) framework, which dynamically determines both the necessity and strength of intervention by tracking the internal states of the LLMs during generation, considering both the question and the generated content. Since intervening after detecting a deviation from the desired behavior is often too late, we further propose the backtracking mechanism to correct the deviated tokens and steer the LLMs toward the desired behavior. Extensive experiments on the TruthfulQA dataset and six multiple-choice datasets demonstrate that our method outperforms baselines. Our code will be released at https://github.com/gjw185/FASB.

大模型对齐激活调制生成控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。