用区块链公开评测开源大模型公平性,确保结果可验证可追溯。
A Transparent Fairness Evaluation Protocol for Open-Source Language Model Benchmarking on the Blockchain
- 基于ICP区块链执行智能合约,自动调用Hugging Face接口并上链存储数据与指标。
- 在PISA、StereoSet和Kaleidoscope数据集上评测Llama等模型,发现跨语言偏差。
- 代码与结果全开源,支持社区审计和模型版本的长期公平性追踪。
大语言模型在刑事司法、教育、医疗和金融等高风险领域广泛应用,但公平性问题仍受关注。本文提出一种基于Internet Computer Protocol(ICP)区块链的透明公平性评估协议,通过智能合约执行链上HTTP请求调用托管在Hugging Face的模型,并将数据集、提示词和评估指标直接上链存储,确保评估过程可验证、不可篡改且可复现。我们在PISA数据集(OECD, 2018)上对Llama、DeepSeek和Mistral模型进行学术表现预测评估,采用统计均等性和等机会度量标准;同时利用StereoSet数据集(Nadeem et al., 2020)的结构化上下文关联指标检测社会偏见。进一步在英文、西班牙语和葡萄牙语中开展多语言评估,使用Kaleidoscope基准(Salazar et al., 2025),揭示跨语言不公平现象。所有代码与结果均开源,支持社区审计与模型版本的长期公平性追踪。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed in realworld applications, yet concerns about their fairness persist especially in highstakes domains like criminal justice, education, healthcare, and finance. This paper introduces transparent evaluation protocol for benchmarking the fairness of opensource LLMs using smart contracts on the Internet Computer Protocol (ICP) blockchain (Foundation, 2023). Our method ensures verifiable, immutable, and reproducible evaluations by executing onchain HTTP requests to hosted Hugging Face endpoints and storing datasets, prompts, and metrics directly onchain. We benchmark the Llama, DeepSeek, and Mistral models on the PISA dataset for academic performance prediction (OECD, 2018), a dataset suitable for fairness evaluation using statistical parity and equal opportunity metrics (Hardt et al., 2016). We also evaluate structured Context Association Metrics derived from the StereoSet dataset (Nadeem et al., 2020) to measure social bias in contextual associations. We further extend our analysis with a multilingual evaluation across English, Spanish, and Portuguese using the Kaleidoscope benchmark (Salazar et al., 2025), revealing cross-linguistic disparities. All code and results are open source, enabling community audits and longitudinal fairness tracking across model versions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。