让非技术人员也能安全评估大模型真实性和责任性。
LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
- 浏览器直通的开源框架，分角色支持技术、领域、合规三类用户
- 本地运行核心指标，外部评估需手动授权，杜绝数据外泄
- 可视化不确定度、多评委共识、三重检索校验，精准识别幻觉
评估大语言模型输出是否具备事实依据、认知校准和方法可复现性，是负责任AI部署的前提。然而现有审计工具对非技术人员不友好：需编程能力且环境配置复杂，云端平台更会将评估数据传至外部服务，使领域专家与合规人员难以参与。我们提出 LLM-FACETS（LLM 事实性跨评估系统）：一个开源、浏览器可访问、插件式架构的框架，围绕技术专家、领域专家、合规人员三类角色设计，对应欧盟AI法案与NIST AI风险管理框架中的利益相关方。数据流清晰可控：确定性指标（BLEU、ROUGE、BERTScore）完全在自托管服务器内运行，无外发传输；LLM评判指标仅在用户明确授权下连接外部API，凭证由用户全程掌控。通过三项机制实现透明化：逐标记概率可视化认知不确定性，多评委共识降低评判偏差，RAG三元组指标（忠实性、答案相关性、上下文相关性）定位幻觉。插件结构支持任意新指标或数据集无缝集成，无需修改评估流程。开放源码版本支持多指标交叉验证，确保可复现性，并将问责权从模型开发团队中剥离。通过18项指标与基准库的交叉验证，确认了框架有效性。
原文摘要 · Abstract (English)
Assessing whether Large Language Models outputs are factually grounded, epistemically calibrated, and methodologically reproducible is a prerequisite for responsible AI deployment. Yet auditing LLMs remains inaccessible to non-technical practitioners: existing tools require programming expertise and non-trivial environment setup, and cloud-hosted platforms transmit evaluation data to external services, creating barriers for domain experts and compliance officers legally responsible for AI oversight. We introduce LLM-FACETS (LLM FActuality Cross-EvaluaTion System): an open-source framework with a browser-accessible interface and a plugin architecture, structured around three practitioner profiles (technical experts, domain experts, compliance officers) that mirror the stakeholder categories identified in the EU AI Act and the NIST AI Risk Management Framework. The architecture makes data flows explicit: deterministic metrics (BLEU, ROUGE, BERTScore) run entirely within the self-hosted server with no outbound transmission; LLM-judge metrics contact external APIs explicitly, with users retaining full credential control. The framework operationalizes transparency through three mechanisms: token-level log-probability visualization for epistemic uncertainty, multi-judge consensus to mitigate judge bias, and RAG Triad metrics (Faithfulness, Answer Relevance, Context Relevance) to detect and localize hallucinations. A plugin architecture allows any new metric or dataset to be integrated without modifying the evaluation pipeline. The open-source implementation enables cross-checking across multiple metrics targeting the same property, ensuring reproducibility and decoupling AI accountability from the teams building the systems assessed. We verify the framework through cross-validation of 18 metric implementations against canonical reference libraries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。