让大模型在生成时自我检查,关键时刻自动纠错。
Self-Reflective Generation at Test Time
- 生成时动态识别不确定的词元,触发自我修正
- 用已生成上下文训练修正向量,提升判断可靠性
- 无需训练即可插件式使用,适合各类大模型
大型语言模型通过长链推理解决复杂任务,但其单向自回归生成过程易受早期错误影响。现有自我反思方法或需重写完整草稿,或依赖昂贵训练,均属被动且低效。为此,本文提出测试时自我反思生成(SRGen),一种轻量级框架,在不确定点生成前进行反思。生成过程中,通过动态熵阈值识别高不确定性词元,为每个识别出的词元训练特定修正向量,利用已有上下文对词元概率分布进行自我反思式修正。通过回溯分析部分输出,该机制显著降低高不确定性点的出错概率。在多个数学推理基准和多种LLM上评估表明,SRGen可有效增强模型推理能力。结果表明,SRGen是即插即用的可靠推理增强方法,具备可控开销,可与其它训练时(如RLHF)和测试时(如SLOT)技术结合。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly solve complex reasoning tasks via long chain-of-thought, but their forward-only autoregressive generation process is fragile; early token errors can cascade, which creates a clear need for self-reflection mechanisms. However, existing self-reflection either performs revisions over full drafts or learns self-correction via expensive training, both fundamentally reactive and inefficient. To address this, we propose Self-Reflective Generation at Test Time (SRGen), a lightweight test-time framework that reflects before generating at uncertain points. During token generation, SRGen utilizes dynamic entropy thresholding to identify high-uncertainty tokens. For each identified token, it trains a specific corrective vector, which fully exploits the already generated context for a self-reflective generation to correct the token probability distribution. By retrospectively analyzing the partial output, this self-reflection enables more trustworthy decisions, thereby significantly reducing the probability of errors at highly uncertain points. Evaluated on challenging mathematical reasoning benchmarks and a diverse set of LLMs, SRGen can significantly strengthen model reasoning. Moreover, our findings position SRGen as a plug-and-play method that integrates reflection into the generation process for reliable LLM reasoning, achieving consistent gains with bounded overhead and can be combined with other training-time (e.g., RLHF) and test-time (e.g., SLOT) techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。