arXiv:2605.18827cs.IRcs.LG2026-05

用可执行代码增强小模型,让其在多选题上表现更好

Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds

论文配图:Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds
图 1 · 摘自论文原文
  • 设计可运行的代码支架,让小模型通过编程推理答题
  • 代码辅助使准确率提升28.1个百分点,最高达66.21%
  • 适合研究小模型推理能力与工具调用效果的研究者

传统多选题评测将小语言模型视为直接作答者,但实际应用中系统常依赖外部工具、代码和多次调用。本文提出代码引导推理(CGR)评估框架及配套可执行程序资源,用于衡量代码支架对小模型在多选题任务中的提升效果。CGR标准化六项组件:统一题目接口、直接求解提示、生成提示、Python支架、求解调用与提取辅助、三通道结果记录。基于本地构建的20,498条保留结果与六种注册求解模型,在非零基线部分,辅助准确率达66.21%,直接准确率为38.11%,差距为+28.10个百分点(双样本置换置信区间[20.32, 36.43])。在更严格要求直接信号>30%的条件下,差距缩小至+14.11点。结果具有描述性,辅助推理消耗更多调用预算,答案提取易出错,Time-MQA存在性能下降,部分生成程序违反禁止硬编码指令。CGR提供完整追踪包,包含直接/辅助/生成侧答案、分区定义、生成代码、响应元数据与审计信息。

原文摘要 · Abstract (English)

Multiple-choice QA benchmarks usually evaluate small language models (SLMs) as direct answerers, but deployed language-model systems increasingly rely on external scaffolds such as tools, code, and repeated model calls. We introduce Code-Guided Reasoning (CGR), an evaluation protocol and generated-program resource for measuring when executable reasoning scaffolds improve SLM performance on MCQA tasks. CGR standardizes six components: a normalized item interface, a direct solver prompt, a generator prompt, a Python scaffold, solver-call and extraction helpers, and a three-channel result record. On 20,498 retained result rows from a locally prepared MCQA bundle and six metadata-registered solver models, the observed non-zero-baseline partition shows 66.21% macro assisted accuracy versus 38.11% direct accuracy, a +28.10 percentage-point difference with a pair-bootstrap interval of [20.32, 36.43]. Under a stricter Ab > 30% direct-signal gate, the macro difference is +14.11 points. These estimates are descriptive. Assisted inference uses a larger solver-call budget, answer extraction is brittle, Time-MQA contains the observed regressions, and some generated programs violate the no-hard-coding instruction. CGR provides the trace package needed to interpret these results, including direct, assisted, and generator-side answers, partition definitions, generated programs, response metadata, and audits.

小模型代码推理多选题评估框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。