开源工具库，一键检测大模型回答是否说谎
TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs
- 整合30多种谎言检测方法，支持黑盒白盒等多种模式
- 在三个数据集上验证，最高准确率超85%
- 适合研究可信生成、AI安全的开发者和研究人员
生成式大语言模型不可避免会产生不实回应。准确预测其输出的真实性至关重要，尤其在高风险场景中。为加速该领域研究并提升谎言检测方法的可及性，我们推出TruthTorchLM——一个开源的综合性Python库，包含30余种真相预测方法（称作Truth Methods）。与仅聚焦文档验证的Guardrails或仅限基于不确定性的LM-Polygraph不同，TruthTorchLM覆盖了计算成本、访问级别、文档依赖和监督类型等多样权衡。该库无缝兼容HuggingFace与LiteLLM，支持本地部署与API模型。提供统一接口用于生成、评估、校准及长文本真相预测，并具备灵活扩展新方法的框架。我们在TriviaQA、GSM8K和FactScore-Bio三个数据集上对代表性方法进行了评估，代码已开源。
原文摘要 · Abstract (English)
Generative Large Language Models (LLMs)inevitably produce untruthful responses. Accurately predicting the truthfulness of these outputs is critical, especially in high-stakes settings. To accelerate research in this domain and make truthfulness prediction methods more accessible, we introduce TruthTorchLM an open-source, comprehensive Python library featuring over 30 truthfulness prediction methods, which we refer to as Truth Methods. Unlike existing toolkits such as Guardrails, which focus solely on document-grounded verification, or LM-Polygraph, which is limited to uncertainty-based methods, TruthTorchLM offers a broad and extensible collection of techniques. These methods span diverse tradeoffs in computational cost, access level (e.g., black-box vs white-box), grounding document requirements, and supervision type (self-supervised or supervised). TruthTorchLM is seamlessly compatible with both HuggingFace and LiteLLM, enabling support for locally hosted and API-based models. It also provides a unified interface for generation, evaluation, calibration, and long-form truthfulness prediction, along with a flexible framework for extending the library with new methods. We conduct an evaluation of representative truth methods on three datasets, TriviaQA, GSM8K, and FactScore-Bio. The code is available at https://github.com/Ybakman/TruthTorchLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。