验证两种大模型推理可靠性方法,发现效果有限且难以复现。
Reproducing and Stress-Testing Two Approaches to LLM Reasoning Reliability: Test-Time Probability Aggregation and Logic-Representation Editing
- 用概率聚合与逻辑编辑提升推理可信度
- 新任务域测试中优势不显著,部分结果还反转
- 代码未公开,结果在不同模型上不稳定
我们独立复现了两项近期提升大语言模型(LLM)推理可靠性的方法,并在多个领域和模型上进行压力测试(RPC在四个新任务域使用Qwen3-8B,LCF在四个7-8B模型上)。第一种方法RPC在推理时聚合词元概率并结合自一致性;第二种方法LCF通过训练投影器将隐藏状态分解为“内容”与“逻辑”并编辑逻辑部分至有效区域。验证此类可靠性声称至关重要,因为原评估均由作者自行完成,从未被独立复现或跨模型/领域测试,且LCF未公开代码。我们重跑RPC的发布路径聚合,重新实现LCF的投影器、对比学习及干预流程,并扩展至文本到SQL、法律信息抽取、谬误识别和判例评分。结果:RPC在原作者释放的推理路径上完全复现原始网格;在四个新任务域中其优于自一致性的表现均不显著(配对p ≥ 0.28),仅在BIRD任务中预算增加时性能随K上升,但当样本量扩大至n=200时,最大增益(+2.5准确率,K=32,p=0.16)转为-0.25。LCF的逻辑有效性方向真实但微弱(最优子层分离度0.82,对比语义属性控制的0.95);其唯一正面影响(Qwen3 $Δ$Prob)不显著(p=0.56),且在另两个模型上显著降低$Δ$Prob。
原文摘要 · Abstract (English)
We independently reproduce two recent methods for making large language model (LLM) reasoning more reliable, and stress-test them across domains and models (RPC across four new task domains with Qwen3-8B, LCF across four 7-8B models). The first, RPC, aggregates token probabilities and self-consistency at inference; the second, LCF, trains projectors that split hidden states into "content" and "logic" and edits the logic part toward a valid region. Validating such reliability claims matters because the original evaluations are run by each method's own authors and were never independently reproduced or stress-tested across models and domains, and LCF shipped no public code. We re-run RPC's published-path aggregation and re-implement LCF's projector, contrastive, and intervention pipeline, then extend both to text-to-SQL, legal extraction, fallacy identification, and precedent grading, and probe LCF's representation directly. RPC reproduces the original grid exactly on the authors' released reasoning paths; on four new domains its edge over self-consistency is never significant (ties or small mixed differences, paired p >= 0.28), and on BIRD, the one domain where we vary the budget, the edge grows with K as predicted but its largest gap (+2.5 accuracy at K=32, p=0.16) reverses to -0.25 when we enlarge the sample to n=200. LCF's logic-validity direction is real but weak (0.82 separability at the single best sub-layer versus 0.95 for a semantic-attribute control); its one positive effect (Qwen3 $Δ$Prob) is not significant (p=0.56), while it significantly reduces $Δ$Prob on two of the other three models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。