不同推理后端可显著影响大模型评测分数,甚至改变模型表现。
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

- 通过跨框架对比实验,发现后端工具影响模型输出
- 39%的评测结果差异源于后端选择,非采样噪声或参数配置
- 建议公开后端及生成参数,用确定性解码做公平比较
评测分数常被视为模型自身属性,但生成这些分数所用的推理框架(如HuggingFace、vLLM、Ollama)通常被认为无影响,且其名称和版本几乎从不披露。本文在全交叉实验中(3个指令微调模型 × 5个推理框架 × 6个基准测试 × 4种生成模式)研究不同工具对评测分数的影响及其与生成超参数的关系。结果表明,后端是不可忽略的因素:即使在贪婪解码、无采样噪声条件下,更换后端仍会显著改变模型表现,且该效应具有结构性并高度依赖模型。按生成模式分解方差发现,实践者观察到的约39%变异来自后端,其余源于采样噪声和各框架默认参数,后者可通过披露与匹配配置避免。这一差异在事实类基准上比社会偏见类更明显。总体而言,评测分数并非后端无关,因此我们建议披露后端、版本及完整生成配置,并使用确定性解码进行跨后端比较。
原文摘要 · Abstract (English)
Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39\%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。