让小模型也能准确问答结构化数据，靠的是自我纠错的蒸馏方法。
Self-Correction Distillation for Structured Data Question Answering
- 用错误提示机制检测生成问题中的错误并反馈修正信息。
- 通过两阶段蒸馏，将大模型的查询生成与纠错能力迁移到小模型。
- 在5个基准上表现优于其他蒸馏方法，接近GPT-4性能，适合资源受限场景。
结构化数据问答（包括表格问答、知识图谱问答和时间知识图谱问答）是重要研究方向。大语言模型的进步推动了统一结构化问答框架如TrustUQA的发展。然而，当应用于小规模模型时，这些框架面临挑战，因为小模型在生成结构化查询时容易出错。为提升小规模模型的结构化数据问答能力，本文提出自纠正蒸馏（SCD）方法。SCD设计了错误提示机制（EPM），在推理过程中检测错误并提供定制化错误提示；同时采用两阶段蒸馏策略，将大模型的查询生成与错误纠正能力迁移至小模型。在5个基准、3种结构化数据类型上的实验表明，SCD在80亿参数的小模型上表现最佳，泛化能力优越，部分数据集上已接近GPT-4性能。此外，配备EPM的大模型在多数数据集上超越现有最优结果。
原文摘要 · Abstract (English)
Structured data question answering (QA), including table QA, Knowledge Graph (KG) QA, and temporal KG QA, is a pivotal research area. Advances in large language models (LLMs) have driven significant progress in unified structural QA frameworks like TrustUQA. However, these frameworks face challenges when applied to small-scale LLMs since small-scale LLMs are prone to errors in generating structured queries. To improve the structured data QA ability of small-scale LLMs, we propose a self-correction distillation (SCD) method. In SCD, an error prompt mechanism (EPM) is designed to detect errors and provide customized error messages during inference, and a two-stage distillation strategy is designed to transfer large-scale LLMs' query-generation and error-correction capabilities to small-scale LLM. Experiments across 5 benchmarks with 3 structured data types demonstrate that our SCD achieves the best performance and superior generalization on small-scale LLM (8B) compared to other distillation methods, and closely approaches the performance of GPT4 on some datasets. Furthermore, large-scale LLMs equipped with EPM surpass the state-of-the-art results on most datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。