用纠错码思路提升大模型生成可靠性,小模型也能超大模型。
SERC: LDPC-Inspired Semantic Error Correction for Retrieval-Augmented Generation

- 借鉴低密度奇偶校验码,只验证关键信息点,降低检查成本。
- 在长文本和真相测试集上,事实准确率提升显著,小模型超越大模型。
- 无需训练、适配任何模型,适合算力有限的场景使用。
尽管大语言模型展现出强大能力,但幻觉问题严重削弱其可靠性。现有内在自纠错方法常因自我偏见而失效,难以识别自身输出错误。为此,我们提出基于LDPC的语义错误纠正方法(SERC),为大模型幻觉提供理论解释与缓解方案。将文本生成视为语义噪声信道,把生成结果看作被噪声污染的码字。受低密度奇偶校验(LDPC)码启发,SERC采用稀疏验证策略:不全面检查所有事实,而是生成少量低密度验证查询,并与外部证据比对,高效检测并修正错误。我们在LongForm Bio和TruthfulQA两个基准上,使用Llama-3-8B和Qwen2.5-14B进行评估。实验表明,SERC优于主流自纠错方法和强基线检索增强模型,在事实准确率(FactScore)方面表现突出。尤其值得注意的是,小型语言模型(SLMs)经SERC后可超越大型基线模型在幻觉抑制与信息保留方面的表现。研究证明,SERC是一种无需训练、模型无关的解决方案,相比密集验证方法大幅降低验证开销,实现资源受限环境下的成本与保真度最优平衡。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) have demonstrated remarkable capabilities, their reliability is significantly compromised by hallucinations. Existing intrinsic self-correction methods attempt to address this, but often fail due to self-bias, where models struggle to identify errors in their own outputs without external verification. To overcome these limitations, we propose the LDPC-inspired semantic error correction for retrieval-augmented generation (SERC), providing a theoretical framework to interpret and mitigate LLM hallucinations. We reformulate the text generation process as a semantic noisy channel, treating generated responses as noise-corrupted codewords. Inspired by low-density parity-check (LDPC) codes, SERC employs a sparse verification strategy: instead of exhaustively checking all facts, it generates low-density verification queries and validates them against external evidence to efficiently detect and correct errors. We evaluate SERC on LongForm Bio and TruthfulQA benchmarks using Llama-3-8B and Qwen2.5-14B. Experimental results demonstrate that SERC outperforms both intrinsic self-correction methods and strong retrieval-augmented baselines, demonstrating significant gains especially in factual precision (FactScore). Notably, SERC enables small language models (SLMs) to surpass the performance of larger baselines in hallucination reduction and information preservation. Our findings demonstrate that SERC provides a training-free, model-agnostic solution that significantly reduces verification overhead compared to dense methods, achieving an optimal trade-off between cost and fidelity in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。