用思维链识别逻辑等价问题，大幅降低大模型评估成本
CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection
- 通过零样本思维链分析，按内在逻辑而非表面文字聚类问题
- 在GSM8K等数据集上评估成本降低90%以上，分数估计误差小于5%
- 适合需要频繁评估复杂推理任务的研究者和开发者
大语言模型的持续评估带来巨大计算开销。现有核心集选择方法或依赖大量历史日志（如项目反应理论），存在严重冷启动瓶颈；或受表层词汇偏差影响，忽略任务背后的推理结构。我们提出CoT-Core，一种无需训练的核心问题选择框架。该方法利用大模型生成零样本思维链（CoT）推理路径，将这些路径投影到隐空间，按内在逻辑等价性而非表面文本相似性对问题聚类。在GSM8K、MMLU、MMLU-Pro和GPQA上的实验表明，CoT-Core显著降低评估成本，同时保持高保真度的得分估计，并揭示了推理感知剪枝的边界条件：其有效性本质上由任务复杂度决定。
原文摘要 · Abstract (English)
Evaluating Large Language Models (LLMs) incurs prohibitive computational overhead during continuous development processes. While coreset selection accelerates evaluation, existing methods either suffer from a severe ``cold start'' bottleneck requiring massive historical logs (e.g., Item Response Theory) or exhibit a surface lexical bias that misses the underlying reasoning manifold of tasks. We propose CoT-Core, a novel training-free core question selection framework. Recognizing that lexically disparate questions can share equivalent underlying logic, CoT-Core prompts LLMs to unroll zero-shot Chain-of-Thought (CoT) reasoning trajectories. Projecting these paths into a latent space effectively clusters questions by intrinsic logical equivalence rather than superficial text similarity. Extensive experiments on GSM8K, MMLU, MMLU-Pro, and GPQA demonstrate that CoT-Core drastically reduces evaluation costs while maintaining high-fidelity score estimation, and delineate the boundary conditions of reasoning-aware pruning, revealing that its efficacy is intrinsically gated by task complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。