无需外部模型，用对比故事对评估稀疏自编码器的可解释性。
CE-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoders
- 基于精心筛选的对比故事对构建评估基准
- 与SAEBench结果相关性超70%，且不依赖外部LLM判断
- 适合研究大模型可解释性与自动评估方法的学者
稀疏自编码器（SAEs）是揭示大语言模型中可解释特征的有前景方法。尽管已有多种自动化评估方式，但多数依赖外部大语言模型进行判断。本文提出CE-Bench，一个轻量级、基于对比故事对的新型对比评估基准，用于衡量SAEs的可解释性。通过全面实验验证，CE-Bench能可靠评估可解释性，与现有基准（如SAEBench）高度一致，相关性超过70%，且无需外部LLM参与。官方代码与数据集已开源，可供公开使用。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) are a promising approach for uncovering interpretable features in large language models (LLMs). While several automated evaluation methods exist for SAEs, most rely on external LLMs. In this work, we introduce CE-Bench, a novel and lightweight contrastive evaluation benchmark for sparse autoencoders, built on a curated dataset of contrastive story pairs. We conduct comprehensive evaluation studies to validate the effectiveness of our approach. Our results show that CE-Bench reliably measures the interpretability of sparse autoencoders and aligns well with existing benchmarks without requiring an external LLM judge, achieving over 70% Spearman correlation with results in SAEBench. The official implementation and evaluation dataset are open-sourced and publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。