发现大模型因精度不同导致输出分歧,提出自动检测方法
Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements
- 设计差分测试框架PrecisionDiff,生成敏感输入检测精度差异
- 实测多模型多精度下普遍存在行为分歧,部分漏洞可被利用
- 适合模型开发者与安全评估人员用于提升精度鲁棒性
大型语言模型(LLMs)在部署时常采用不同数值精度配置,如bfloat16、float16及int16、int8等量化格式,以平衡效率与资源消耗。然而,不同精度间的细微差异难以察觉,现有评估方法常忽略此类问题。本文提出PrecisionDiff,一种自动化差分测试框架,通过生成精度敏感测试用例并进行跨精度对比分析,系统性地揭示传统测试策略下隐藏的行为分歧。以对齐验证任务为例,实验发现精度差异会导致越狱式分歧:某些输入在一种精度下被拒绝,却在另一精度下生成有害响应。结果表明,此类行为分歧在多个开源对齐LLM及多种精度设置中广泛存在,且PrecisionDiff显著优于常规测试方法。本工作实现了精度敏感测试用例的自动化生成,支持部署前有效评估,助力训练阶段提升精度鲁棒性。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。