通过微调模型权重比特,实现大模型无限生成,大幅增加推理成本。
BitHydra: Towards Bit-flip Inference Cost Attack against Large Language Models
- 通过比特翻转扰动模型权重,不依赖输入即可触发高成本生成。
- 仅需1-4个比特翻转,即可让10个不同规模模型陷入无限生成。
- 采用连续优化方法解决离散搜索难题,对防御机制也有效。
大型语言模型(LLMs)广泛部署,但其高昂的计算开销使其易受推理成本攻击,此类攻击旨在故意最大化输出长度。本文研究了一种新攻击面:通过篡改模型参数而非输入来最大化推理成本。该方法利用比特翻转攻击(BFAs)可通过对权重进行微小扰动持久改变模型行为的能力,从而将攻击与特定输入查询解耦。为此,我们提出BitHydra框架,解决识别使生成成本最大化的精确权重比特这一独特优化挑战。我们将攻击建模为约束二元整数规划(BIP)问题,系统性地抑制结束序列(<eos>)概率。为克服离散搜索空间的不可行性,将问题松弛为连续优化任务,并通过交替方向乘子法(ADMM)求解。我们在10个不同规模的LLM(1.5B–16B)上评估了BitHydra。结果表明,所提优化方法能高效实现无限生成,且仅需1-4个比特翻转即可在所有测试模型上达成,验证了基于ADMM的公式对标准模型及潜在防御的有效性。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely deployed, but their substantial compute demands make them vulnerable to inference cost attacks that aim to deliberately maximize the output length. In this work, we investigate a distinct attack surface: maximizing inference cost by tampering with the model parameters instead of inputs. This approach leverages the established capability of Bit-Flip Attacks (BFAs) to persistently alter model behavior via minute weight perturbations, effectively decoupling the attack from specific input queries. To realize this, we propose BitHydra, a framework that addresses the unique optimization challenge of identifying the exact weight bits that maximize generation cost. We formulate the attack as a constrained Binary Integer Programming (BIP) problem designed to systematically suppress the end-of-sequence (i.e., <eos>) probability. To overcome the intractability of the discrete search space, we relax the problem into a continuous optimization task and solve it via the Alternating Direction Method of Multipliers (ADMM). We evaluate BitHydra across 10 LLMs (1.5B-16B). Our results demonstrate that the proposed optimization method efficiently achieves endless generation with as few as 1-4 bit flips on all testing models, verifying the effectiveness of the ADMM-based formulation against both standard models and potential defenses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。