改进RAG微调方法，让模型生成技术文档更准确可靠
Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation
- 用新评估框架分离模型参数知识与外部信息来源
- 7B微调模型性能媲美甚至超过72B大模型
- 发现微调会减少幻觉但不增强正确参数知识
检索增强生成（RAG）微调在问答任务中表现优异，但其能否迁移到专业领域文档生成仍未知。本文研究电子工程领域需求文档生成中的监督式RAG微调（RAG-SFT），对两个7B模型采用不同训练数据策略。由于Rouge和BertScore在长技术文本上难以衡量事实性，提出基于声明的评估流程C-FEX，可追踪每条生成内容的来源（增强提示或参考回答），并定义参数知识精确率（PKP），用于衡量模型权重中生成内容的正确性。实验证明，原有参数知识评估指标可分解为PKP × PR，分离出参数输出频率（PR）与质量（PKP）。结果表明：微调后的7B模型性能达到甚至超越72B基线；标准指标与声明级事实性评估结论不一致，可能误导对微调效果的判断；微调并未增强正确参数知识，反而抑制了幻觉——模型使用自身参数的频率降低，但输出更可靠。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) fine-tuning has shown substantial improvements over vanilla RAG, yet most studies target document question answering, leaving open whether these gains transfer to specialized tasks. We study supervised RAG fine-tuning (RAG-SFT) for requirements document generation in the electronics engineering domain, adapting two 7B models under two different training data strategies. Because Rouge and BertScore poorly capture factuality on long technical text, we introduce C-FEX, a claim-based evaluation pipeline that attributes each response claim to its origin (augmented prompt or reference response), and propose Parametric Knowledge Precision (PKP), which isolates claims originating from the model's weights and measures their correctness. We show that a prior metric to assess parametric knowledge decomposes as PKP $\times$ PR, separating the rate of parametric output (PR) from its quality (PKP). Empirically, fine-tuned 7B models match or exceed a 72B baseline; standard metrics disagree with claim-based factuality and can mislead about fine-tuning gains; and, fine-tuning does not reinforce correct parametric knowledge but suppresses hallucination---models speak from their weights less often but far more reliably.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。