用大模型自我修正推理链,低成本训练小模型解决数据少的复杂任务
Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correction
- 让大模型逐步检查并修正小模型的推理过程,生成高质量监督信号
- 在600个数据库图谱任务中,准确率提升显著,错误漏检率降低47%
- 适合教育、医疗等标注数据稀缺领域的模型定制化开发
当特定任务缺乏直接人类标注或高质量标签时,训练专用的小型推理模型面临挑战。然而,具备推理能力的大模型能生成大量中间推理轨迹,这些轨迹可系统性地被优化以形成有效的监督信号。本文提出R2tA方法,将优化后的模型推理链转化为训练专用推理模型的监督数据。首先,基于开源基础模型对特定任务输入生成初始推理与回答;随后,通过多步评估与修正,消除幻觉与不一致,构建高保真数据集。采用两阶段对齐策略:先进行监督微调(SFT),再通过直接偏好优化(DPO)使模型推理符合人工验证的概念偏好,并据此校准最终输出。以数据库设计中的扩展实体关系图(EERDs)为案例,该任务结构复杂,仅靠提示易遗漏或虚构错误。我们构建了包含600个变体的数据集(训练/测试划分450/150),涵盖11类人为引入的错误。实证表明,R2tA为数据稀缺领域提供了可扩展、低成本的LLM适配路径,支持教育等领域可复现的AI工具开发。
原文摘要 · Abstract (English)
Training a task-specific small reasoning model is challenging when direct human supervision or high-quality labels are scarce. However, LLMs with reasoning capabilities produce abundant intermediate reasoning traces that can be systematically refined to create effective supervision signals. We propose Reason-Refine-then-Align (R2tA), which turns refined model rationales into supervision for training task-specific reasoning models. Our method generates initial reasoning and responses from an open-source base model on task-specific inputs, then refines these traces, fixing hallucinations and inconsistencies, to form a high-fidelity dataset. We perform a two-stage alignment, supervised fine-tuning (SFT), followed by direct preference optimization (DPO) to calibrate the model's intermediate reasoning with human-validated conceptual preferences and then condition the final output on that aligned reasoning. As a case study, we apply R2tA to evaluate extended entity relationship diagrams (EERDs) in database system design, a structurally complex task where prompt-only methods miss or hallucinate errors. We curated a dataset of 600 EERD variants (train/test split of 450/150, respectively) with induced mistakes spanning 11 categories. Empirical evaluation suggests R2tA provides a practical, cost-effective path to scalable LLM adaptation in data-scarce domains, enabling reproducible AI tools for education and beyond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。