研究提示工程对药物毒性预测的影响,发现模型变异性远超提示微调效果。
Analysis of Prompt Engineering for Drug Toxicity Prediction

- 通过设计不同提示角色、结构和规则,分析其对LLM输出的影响。
- 使用化学信息学特征提取使模型性能显著优于LLM生成值。
- 方法可推广至生物信息学多个领域,适合药物研发与AI交叉研究者。
英国临床试验成本可达130万英镑,约90%的药物因失败而终止,毒性是主要原因之一,且测试耗时耗力。近年来,人工智能特别是大语言模型(LLMs)被广泛探索用于药物毒性预测。然而,LLMs在提示微调时表现出显著敏感性,提示工程可能影响结果一致性。本文提出一种分析方法,探究提示措辞在药物毒性预测中的重要性。通过设置不同任务角色、提示结构与规则解释方式,引导LLMs识别预测毒性相关的化学特性,并基于这些特征生成数据集,再交由机器学习算法处理。实验表明,LLMs固有的自然变异性远超过提示优化带来的改进;但若采用化学信息学代码提取特征,而非依赖LLM生成值,则模型性能有显著提升。该分析框架适用于多种提示类型,在生物信息学多个领域具有普适性。
原文摘要 · Abstract (English)
Clinical trials in the UK can cost up to {\pounds}1.3 million, with approximately 90% drug failure rate. Toxicity is a major contributing factor in drug failure. Testing is time and cost intensive. In recent years, the use of artificial intelligence has been increasingly explored to aid in the prediction of drug toxicity, with extensive use of large language models (LLMs). However, LLMs can show considerable variation when minor changes are made to prompts, which raises concerns about their sensitivity to prompt engineering. Prompt engineering is used to optimise a prompt given to an LLM to generate the desired output. This paper proposes a method to analyse prompt engineering for drug toxicity prediction. The aim of the paper is to investigate the importance of prompt phrasing for drug toxicity prediction. LLMs were prompted to identify chemical properties of significance when predicting drug toxicity. Prompts were constructed to investigate; job role, prompt structuring, and rule interpretation. LLMs were then used to generate datasets, using the identified features from initial prompting, which were then passed to machine learning algorithms. The experiments show that the natural variance which occurs in LLMs outweighs any fine-tuning of prompts. There were, however, substantial improvements in model performance when using chemoinformatic code to extract features instead of using LLM-generated values. The proposed analysis methodology is applicable to a wide range of prompt types across different areas of bioinformatics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。