将大模型推理过程拆解为概念图谱，提升小模型训练效率与可解释性。
Distilling LLM Reasoning into Graph of Concept Predictors
- 用有向无环图建模教师模型的推理路径，学生端对应模块化概念预测器。
- 在8个文本分类任务上，少标注数据下性能优于传统蒸馏方法。
- 支持定位错误源头，适合需要可解释性的高可靠性应用场景。
将大型语言模型（LLMs）部署于判别性任务常受推理延迟、计算开销和大规模API成本限制。主动蒸馏通过查询LLM专家来训练紧凑的判别性学生模型，以降低这些开销，但多数流程仅蒸馏最终标签，丢弃中间推理信号，难以诊断推理缺失或错误发生位置。本文提出概念预测图（Graph of Concept Predictors, GCP），一种感知推理的主动蒸馏框架，将教师模型的决策过程外化为有向无环图，并在学生端用模块化概念预测器镜像该结构。GCP通过图感知的样本选择策略，聚焦关键推理节点的不确定性与分歧，提升样本效率；同时通过针对性子模块重训，将下游损失归因于特定概念预测器，仅更新影响最大的模块，从而增强训练稳定性和效率。在八个NLP分类基准上的实验表明，GCP在有限标注预算下提升了性能，且训练过程更具可解释性和可控性。代码已开源：https://github.com/Ziyang-Yu/GCP。
原文摘要 · Abstract (English)
Deploying Large Language Models (LLMs) for discriminative workloads is often limited by inference latency, compute, and API costs at scale. Active distillation reduces these costs by querying an LLM oracle to train compact discriminative students, but most pipelines distill only final labels, discarding intermediate reasoning signals and offering limited diagnostics of what reasoning is missing and where errors arise. We propose Graph of Concept Predictors (GCP), a reasoning-aware active distillation framework that externalizes the teacher's decision process as a directed acyclic graph and mirrors it with modular concept predictors in the student. GCP enhances sample efficiency through a graph-aware acquisition strategy that targets uncertainty and disagreement at critical reasoning nodes. Additionally, it improves training stability and efficiency by performing targeted sub-module retraining, which attributes downstream loss to specific concept predictors and updates only the most influential modules. Experiments on eight NLP classification benchmarks demonstrate that GCP enhances performance under limited annotation budgets while yielding more interpretable and controllable training dynamics. Code is available at: https://github.com/Ziyang-Yu/GCP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。