通过操控模型内部概念，揭示大模型在高风险任务中对敏感属性的隐性依赖。
White-Box Sensitivity Auditing with Steering Vectors
- 利用激活操控技术，在模型内部直接测试关键概念的影响。
- 在四类模拟高风险任务中发现模型严重依赖性别等敏感属性。
- 适合关注模型公平性、可解释性的研究人员和产品团队。
算法审计是检测系统是否满足监管要求或运营期望的重要工具。当前大语言模型（LLM）的审计主要依赖黑箱评估，仅通过输入输出测试判断模型行为，且测试样本多基于启发式生成，局限于输入空间。此外，许多社会相关属性（如性别偏见）抽象难测，仅靠文本输入难以准确衡量。为此，我们提出一种白盒敏感性审计框架，利用激活操控技术，从模型内部开展更严格的评估。该方法通过操纵与任务目标相关的关键概念，进行内部敏感性测试。我们在四个模拟高风险的LLM决策任务中验证了该方法的有效性，结果表明模型预测仍显著依赖受保护属性，即使标准黑箱评估显示无明显偏见。代码已开源：https://github.com/hannahxchen/llm-steering-audit。
原文摘要 · Abstract (English)
Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently indicates substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。