首个聚焦思维链蒸馏的数据驱动评测基准，助力高效推理模型优化
The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
- 构建多视角数据操控评测框架，系统评估数据增强、筛选与混合效果
- 在多种教师模型与学生架构上验证，显著提升小模型在跨域场景的推理能力
- 提供可复现的实践指南，适合追求高效推理模型的研究者参考
以数据为中心的蒸馏技术（包括数据增强、选择和混合）为构建更小、更高效的推理型学生大模型提供了新路径。然而，当前缺乏系统性评估各类蒸馏方法的综合性基准。本文提出首个数据驱动的思维链蒸馏评测基准DC-CoT，从方法、模型与数据三个维度，系统研究数据操作对思维链蒸馏的影响。基于多个教师模型（如o4-mini、Gemini-Pro、Claude-3.5）与学生架构（如3B、7B参数），在多个推理数据集上，全面评估不同数据策略对学生模型性能的影响，重点关注分布内（IID）与分布外（OOD）泛化能力及跨领域迁移表现。研究结果为优化思维链蒸馏提供可落地的实践经验，推动更高效、可访问的推理模型发展。代码库已开源：https://github.com/UNITES-Lab/Distillation-Bench
原文摘要 · Abstract (English)
Data-centric distillation, including data augmentation, selection, and mixing, offers a promising path to creating smaller, more efficient student Large Language Models (LLMs) that retain strong reasoning abilities. However, there still lacks a comprehensive benchmark to systematically assess the effect of each distillation approach. This paper introduces DC-CoT, the first data-centric benchmark that investigates data manipulation in chain-of-thought (CoT) distillation from method, model and data perspectives. Utilizing various teacher models (e.g., o4-mini, Gemini-Pro, Claude-3.5) and student architectures (e.g., 3B, 7B parameters), we rigorously evaluate the impact of these data manipulations on student model performance across multiple reasoning datasets, with a focus on in-distribution (IID) and out-of-distribution (OOD) generalization, and cross-domain transfer. Our findings aim to provide actionable insights and establish best practices for optimizing CoT distillation through data-centric techniques, ultimately facilitating the development of more accessible and capable reasoning models. The codebase can be accessed at https://github.com/UNITES-Lab/Distillation-Bench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。