低成本评估大模型偏见、毒性与真实性，支持笔记本运行。
TriEval: A Resource-Efficient Pipeline for LLM Bias, Toxicity, and Truthfulness Assessment
- 三合一评估框架，同时检测偏见、毒性与事实性
- 在4个主流模型上验证，开源模型更易产生毒性
- 仅需普通笔记本，无需GPU集群，适合资源有限研究者
大语言模型已广泛应用于医疗、教育和政府服务，其安全性与公平性需持续评估。部署后常见问题包括输出不一致和幻觉。现有评估工具多仅支持单参数测试或依赖大量算力，难以普及。TriEval 提出一种资源高效管道，可同时评估模型在偏见、毒性与真实性三个维度的表现，兼容开源与闭源模型，可在无GPU的普通笔记本上运行。在 Llama 3 8B、Mistral 7B、Gemma 2 9B 和 Claude Haiku 四个模型上测试显示，闭源模型在毒性与真实性方面表现更优。该工具已开源，旨在降低研究门槛。
原文摘要 · Abstract (English)
LLMs have evolved from basic chatbots to the backbone of the AI ecosystem, now widely used in healthcare, schools, and government services. The domain-wide adoption of LLMs necessitates continuous evaluation to ensure their safety and fairness. Common issues encountered after deploying LLMs include inconsistent outputs and hallucinations of incorrect information. Although numerous LLM evaluation tools exist, most are limited to testing a single parameter at a time or require massive computational resources that are not accessible to most researchers. TriEval addresses these challenges by evaluating LLM outputs across multiple parameters, including bias, toxicity, and truthfulness together, while minimizing computing resources. The pipeline is compatible with both open- and closed-source models and runs on a standard laptop without a GPU cluster. TriEval has been tested on four models: Llama 3 8B, Mistral 7B, Gemma 2 9B, and Claude Haiku. The results show clear differences between open-source and closed-source models, especially in terms of toxicity and truthfulness. TriEval is being released as open source to enable broader access for researchers with limited computational resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。