量化医学大模型的置信度可靠性受评分方式与提示设计影响，需优先考虑。
When Calibration Depends on the Scoring Rule: Quantized Biomedical LLM Classification
- 用平均分词对数似然替代求和，显著改变模型校准表现
- 提示模板不同可致准确率变化达17.8个百分点，校准效果反转
- 量化至INT4后性能稳定，但温度缩放仅在特定评分下有效
量化大语言模型可在本地处理敏感数据，但其置信度估计必须可信。可靠性取决于提示模板、标签表述和评分归一化等常被忽视的实现选择。我们评估了三个70亿参数的Mistral变体（基础版、BioMistral、指令微调版）在医学摘要五分类任务中，分别采用FP16、INT8、INT4精度的表现。主模板在n=2,000测试句上评估，辅模板在n=200验证句上测试。因主模板基于100个测试样例选择，且初步评分器存在词边界错误，相关结果为探索性。结果显示，评分归一化方式主导校准表现：从求和转为均值后，BioMistral的平均校准误差几乎翻三倍，而指令微调模型下降超一半，准确率仅变动1.4个百分点以内。负对数似然与Brier得分确认该反转。主模板下，提示设计使准确率波动2.9–17.8个百分点，视模型而定。INT8对专用模型影响≤1.1个百分点；INT4效果混合但未崩溃。事后温度缩放在求和评分下提升校准，但尚未在反转排名的归一化方式下验证。这些探索性结果表明，评分归一化与提示设计应作为解码器分类器校准比较的一阶实验决策。
原文摘要 · Abstract (English)
Quantized large language models enable on-premises processing of sensitive data, but their confidence estimates must be trustworthy. Reliability depends on implementation choices--prompt template, label wording, and scoring normalization--that are seldom treated as experimental variables. We evaluate three 7-billion-parameter Mistral variants (base, BioMistral, and instruction-tuned) at FP16, INT8, and INT4 on five-class sentence classification in medical abstracts. Two primary templates are evaluated on n=2,000 test sentences and two auxiliary templates on n=200 validation sentences. Because the primary templates were selected using 100 test-derived examples and a preliminary scorer later found to have a token-boundary error, results involving them are exploratory. Within this design, candidate-scoring normalization dominates apparent calibration: switching from summed to mean-token log-likelihood reverses which model appears better calibrated (BioMistral's mean calibration error nearly triples, while the instruction-tuned model's drops by more than half), yet accuracy changes by at most 1.4 percentage points for the two specialized models. Negative log-likelihood and Brier score confirm the reversal. Across the two primary templates, prompt choice changes mean accuracy by 2.9--17.8 percentage points, depending on the model. INT8 changes accuracy by at most 1.1 percentage points for the specialized models; INT4 effects are mixed but non-catastrophic. Post-hoc temperature scaling improves calibration under summed scoring but has not been validated under the alternative normalization that reverses the ranking. These exploratory results indicate that scoring normalization and prompt design should be first-order experimental decisions in calibration comparisons of decoder-based classifiers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。