对比19种池化策略，发现层级池化在概念表征上显著优于常用方法。
PoolBench: A Benchmark for Pooling Strategies in Concept Representation Evaluation for Decoder-Only LLMs
- 构建标准化评测基准，单独测试不同池化方法性能。
- 层级池化平均准确率达0.7799，远超传统最后标记法的0.7640。
- 结果适用于各类模型和层，适合关注表征质量的研究者。
池化是解码器仅用模型中概念表征的重要但未受充分研究的设计选择：需将标记级隐藏状态压缩为段落级向量，但缺乏统一的比较协议。现有报告的提升常混杂数据集、层数、构造方法与池化规则的同步变化，难以做出合理决策。本文提出PoolBench，一个将池化作为唯一变量、固定评估协议的基准。涵盖17个概念、19种池化策略及3个开源解码器模型（Llama-3.1-8B、Gemma-2-9B、Mistral-7B），在包含37,693条真实文本段落的单一审计语料上评估。主要指标为线性可分性（D1/AUROC），辅以概念主导性（D2/SCP）与输出解耦性（D3）作为诊断指标。核心发现：W4_hierarchical在跨模型平均AUROC达0.7799，而广泛使用的P1_last_token仅0.7640，且统计显著更差（Friedman+Nemenyi，p=2.0e-36；18种有效策略中77对显著差异）。排名在各层间稳定（rho=0.961–0.990）。关键负结果：强检测不等于强控制——多数概念的D2与D3显著弱于D1，反映根本性表征局限而非池化缺陷。中等难度概念上，W4_hierarchical优于P1_last_token 0.042–0.113 AUROC；构造方法差异（DiffMean vs. REPE）影响（delta AUROC 0.15）大于池化（delta AUROC 0.016），确立实际优先级。我们公开语料、预提取激活、评分模型、控制向量与评估代码，作为可复用的池化研究协议。
原文摘要 · Abstract (English)
Pooling is a consequential but under-examined design choice in decoder-only concept representation work: practitioners must collapse token-level hidden states into a passage-level vector, yet no shared protocol exists for comparing this choice across concepts, models, and tasks. Reported gains are confounded by simultaneous changes in dataset, layer, construction method, and pooling rule, making principled decisions impossible. We introduce PoolBench, a benchmark that isolates pooling as the experimental variable under a fixed evaluation protocol. PoolBench covers 17 concepts, 19 pooling strategies, and 3 open-weight decoder-only models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B), evaluated on a single audited corpus of 37,693 real-text passages. The primary axis is linear separability (D1/AUROC); steered concept prevalence (D2/SCP) and output-level disentanglement (D3) serve as diagnostic axes. The primary finding is decisive: W4_hierarchical reaches a cross-model mean AUROC of 0.7799, while the widely adopted P1_last_token baseline reaches only 0.7640 and is statistically significantly worse (Friedman+Nemenyi, p = 2.0e-36; 77 significant pairs among 18 effective strategies). Rankings are stable across layers (rho = 0.961--0.990). A key negative result: strong detection does not imply strong steering -- D2 and D3 are substantially weaker than D1 for most concepts, indicating a fundamental representational limit rather than a pooling failure. On mid-difficulty concepts, W4_hierarchical outperforms P1_last_token by 0.042--0.113 AUROC; construction method choice (DiffMean vs. REPE) has a larger effect (delta AUROC 0.15) than pooling (delta AUROC 0.016), establishing the correct practical hierarchy. We release the corpus, pre-extracted activations, scorer models, steering vectors, and evaluation code as a reusable protocol for pooling research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。