对比打分是关键,复杂设计未必有用。
When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Design
- 用对比打分方法,以低成本恢复跨文档上下文。
- 改进实现后,0.24升至0.66,验证了核心机制有效。
- 简单拼接最高分句作锚点,效果不输复杂构造。
基于锚点的逐项重排序通过共享参考段落对每个候选进行评分,以较低代价恢复跨文档上下文。我们以GCCP/PAGC为例,开展复现先行的研究。初始复现仅基于论文文本,仅得0.24 nDCG@10,远低于报告的0.66,揭示多个未披露的实现细节至关重要。识别并还原八项关键细节后,结果在1.6%内复现原论文数据,并用于受控分析。发现核心对比打分机制在严格统计校正下仍稳健。但原始论文中两个固定设计选择可靠性较低:当第一阶段检索器为BM25时,对比分数与标准相关性分数结合有帮助;而当使用更强的稠密模型(如E5)时,该结合几乎无增益。此外,原文复杂的锚点构建方法实属冗余,简单通过排序最高句交错构造的锚点,在多个数据集上表现相当或更优。结论在不同LLM骨干模型(包括4比特量化720亿参数模型)上一致。总体而言,锚点重排序的有效性主要来自对比打分,而非复杂聚合与锚点构建策略,且适用条件比原评估所暗示更窄。
原文摘要 · Abstract (English)
Anchor-based pointwise LLM reranking scores each candidate against a shared reference passage to recover cross-document context at pointwise cost. We study when this actually helps, using GCCP/PAGC as a representative method. Our study is reproduction-first. We use reproduction as a starting point for a controlled component-level stress test of anchor-based pointwise reranking. Our initial reimplementation, based only on the paper text, achieves 0.24 nDCG@10 instead of the reported 0.66, revealing that several undocumented implementation details are necessary to reproduce the method. After identifying and recovering eight such details, we reproduce the reported results within 1.6% and use the validated implementation for controlled analysis. We find that the core contrastive scoring idea is robust under rigorous statistical correction. However, two design choices held fixed in the original paper are less reliable. First, we find that combining the contrastive score with the standard pointwise relevance score helps when the first-stage retriever is BM25, but gives little or no benefit when the first-stage retriever is a stronger dense model such as E5. Second, the paper's more complex method for constructing the anchor is unnecessary. A much simpler anchor, built by interleaving the top-ranked sentences, matches or outperforms it across datasets. These findings are consistent across different LLM backbones, including a 4-bit quantized 72B model. Overall, anchor-based pointwise reranking is effective, but its gains come mainly from contrastive scoring rather than from the more complex aggregation and anchor-construction choices, and they appear under narrower conditions than the original evaluation suggests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。