提出统一评估大模型在安全、价值、文化三方面的对齐程度的新基准。
Are Aligned Large Language Models Still Misaligned?
- 构建跨三维度的统一数据集SAVACU,覆盖112个领域38万样本。
- 单维度对齐模型在联合评估中误判率超50%,对齐得分仅63%-66%。
- 适合关注大模型实际应用风险的研究者与开发者参考。
大型语言模型的不对齐问题表现为模型行为偏离人类期望,无法同时满足安全、价值与文化维度的要求,而这些维度在真实场景中必须协同作用以解决实际问题。现有基准如INSECURE CODE(侧重安全)、VALUEACTIONLENS(侧重价值)和CULTURALHERITAGE(侧重文化)均只评估单一维度,无法实现多维联合评估。为此,本文提出Mis-Align Bench,一个统一的多维对齐评估基准。首先,基于LLM-PROMPT-DATASET,利用Mistral-7B-Instruct-v0.3将提示重新分类至14个安全领域、56个价值领域和42个文化领域,构建出包含382,424条样本的SAVACU数据集,并通过Llama-3.1-8B-Instruct与SimHash指纹技术扩充低资源领域以避免重复。其次,采用两阶段拒绝采样为每个提示配对不当齐与对齐响应,确保质量。最后,对通用模型、微调模型及开源权重模型进行系统性评估。实验表明,单维度对齐模型虽覆盖率高达97.6%,但在联合条件下误报率超过50%,对齐得分仅为63%-66%。
原文摘要 · Abstract (English)
Misalignment in Large Language Models (LLMs) arises when model behavior diverges from human expectations and fails to simultaneously satisfy safety, value, and cultural dimensions, which must co-occur in real-world settings to solve a real-world query. Existing misalignment benchmarks-such as INSECURE CODE (safety-centric), VALUEACTIONLENS (value-centric), and CULTURALHERITAGE (culture centric)-rely on evaluating misalignment along individual dimensions, preventing simultaneous evaluation. To address this gap, we introduce Mis-Align Bench, a unified benchmark for analyzing misalignment across safety, value, and cultural dimensions. First we constructs SAVACU, an English misaligned-aligned dataset of 382,424 samples spanning 112 domains (or labels), by reclassifying prompts from the LLM-PROMPT-DATASET via taxonomy into 14 safety domains, 56 value domains, and 42 cultural domains using Mistral-7B-Instruct-v0.3, and expanding low-resource domains via Llama-3.1-8B-Instruct with SimHash-based fingerprint to avoid deduplication. Furthermore, we pairs prompts with misaligned and aligned responses via two-stage rejection sampling to enforce quality. Second we benchmarks general-purpose, fine-tuned, and open-weight LLMs, enabling systematic evaluation of misalignment under three dimensions. Empirically, single-dimension models achieve high Coverage (upto 97.6%) but incur False Failure Rate >50% and lower Alignment Score (63%-66%) under joint conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。