训练模型更偏爱合成技术,通过微调可提升其对生物方案的偏好。
Bioalignment: Measuring and Improving LLM Disposition Toward Biological Systems for AI Safety
- 用50个精心设计提示评估10个模型,发现多数倾向合成技术
- 仅用2200万词生物论文微调,模型对生物方案评分显著提升
- 微调后仍保持通用能力,适合关注AI安全的研究者
大规模语言模型在互联网数据上训练后可能表现出系统性偏差,增加非预期行为风险。本研究考察了材料、能源、制造和算法四个领域中模型对合成与生物技术方案的倾向性。使用50个定制化生物对齐提示,基于类似凯利准则的评估框架,测量了5个前沿模型和5个开源模型。结果显示,多数模型未实现生物对齐,倾向于选择非生物方案。随后,我们测试了微调能否提升两个开源模型(Llama 3.2-3B-Instruct 和 Qwen2.5-3B-Instruct)对生物方案的偏好。使用来自6,636篇PMC文章的约2200万词生物问题解决语料进行微调,其中对Llama采用持续训练与指令格式混合方式,对Qwen仅使用指令格式。结果表明,QLoRA微调显著提升了两个模型对生物方案的评分(校正后p < 0.001 和 p < 0.01),且未损害其通用能力。这说明少量微调即可改变模型对生物与合成方案的权衡。尽管研究聚焦小型开源模型,但方法可扩展至更大模型,有助于开发更倾向生物路径的AI系统。相关基准、语料、代码及适配器权重均已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) trained on internet-scale corpora can exhibit systematic biases that increase the probability of unwanted behavior. In this study, we examined potential biases towards synthetic vs. biological technological solutions across four domains (materials, energy, manufacturing, and algorithms). A sample of 5 frontier and 5 open-weight models were measured using 50 curated Bioalignment prompts with a Kelly criterion-inspired evaluation framework. According to this metric, most models were not bioaligned in that they exhibit biases in favor of synthetic (non-biological) solutions. We next examined if fine-tuning could increase the preferences of two open-weight models, Llama 3.2-3B-Instruct and Qwen2.5-3B-Instruct, for biological-based approaches. A curated corpus of ~22M tokens from 6,636 PMC articles emphasizing biological problem-solving was used first to fine-tune Llama 3B with a mixed corpus of continued training and instruction-formatted. This was then extended to Qwen 3B using instruction-formatted only. We found that QLoRA fine-tuning significantly increased the scoring of biological solutions for both models without degrading general capabilities (Holm-Bonferroni-corrected p < 0.001 and p < 0.01, respectively). This suggests that even a small amount of fine-tuning can change how models weigh the relative value of biological and bioinspired vs. synthetic approaches. Although this work focused on small open-weight LLMs, it may be extensible to much larger models and could be used to develop models that favor bio-based approaches. We release the benchmark, corpus, code, and adapter weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。