提出新方法实现无放回采样下的样本高效重用，提升奖励估计精度。
Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-$K$ Objective
- 基于排名条件的霍夫蒂茨-汤普森估计，实现无放回采样中所有子集的无偏重用。
- 通过动态规划将组合计算压缩为一维积分，降低复杂度至O(n log n + nKQ)。
- 适用于精确随机束搜索等结构化序列生成，特别适合高阶奖励评估场景。
我们研究了耦合目标 J_K^WOR = E_{S ~ PL-WOR_K}[max_{i in S} R_i]：即大小为 K 的普拉克特-卢克无放回抽样下的期望最大奖励，该分布对应于 Gumbel-Top-K / 精确随机束搜索解码。此目标与现有样本重用方法所针对的独立同分布目标 J_K^iid = E[max_{i<=K} R_i] 不同，若在耦合采样下直接使用 i.i.d. 权重会导致有偏估计（三项实例中期望偏差恰好为 (4/5) ∇J_K^WOR）。通用联合得分 REINFORCE 虽无偏但缺乏样本重用。本文贡献在于：对 J_K^WOR 子集总和实例化标准排名条件霍夫蒂茨-汤普森估计——从一个 Gumbel-Top-n 池（n>K）及其观测优先级阈值出发，构建可重用全部 C(n,K) 个嵌套子集的无偏估计器，拥有无偏精确评分函数代理梯度；并引入奖励排序的 Max 特定动态规划，将含 K! 项集合概率的 C(n,K)-项求和精确简化为一维积分。固定 Q 阶数的数值积分代价为 O(n log n + nKQ)，数值精确但非代数精确，未提供 ε-近似率保证。当 n ≥ 2K 时，每个非零阶 K 霍夫蒂茨-汤普森项具有有限二阶矩；在同一条件下，完整代理梯度也具有有限二阶矩（边界紧致性尚待确认）。当 K=1 时，该构造退化为经典优先采样。所有量仅需 n+1 个抽样项的概率值及其可微计算图，因此覆盖由精确随机束搜索采样的有限结构化序列策略。有限 Q 数值界与可数无限支持仍为开放问题。验证代码作为附录文件提供。
原文摘要 · Abstract (English)
We study the coupled objective J_K^WOR = E_{S ~ PL-WOR_K}[max_{i in S} R_i]: the expected maximum reward of a size-K Plackett-Luce draw without replacement, the law of Gumbel-Top-K / Stochastic Beam Search decoding. This estimand differs from the conventional i.i.d. objective J_K^iid = E[max_{i<=K} R_i] targeted by existing sample-reuse Max@K estimators, and reusing their i.i.d. weights under the coupled sampler is provably biased (a closed-form three-item instance gives E[g_iid] = (4/5) grad J_K^WOR exactly; pass@K under the coupled sampler is the binary-reward special case). Generic joint-score REINFORCE is already unbiased for J_K^WOR; what it lacks is sample reuse. Our contribution is to instantiate standard rank-conditioned Horvitz-Thompson estimation for the J_K^WOR subset total: from one Gumbel-Top-n pool (n>K) and its observed priority threshold we build an estimator that reuses all C(n,K) embedded K-subsets, unbiased with an unbiased exact score-function surrogate gradient, plus a reward-sorted Max-specific dynamic program that collapses the C(n,K)-term subset sum (with K!-cost set probabilities) exactly to a one-dimensional integral. A fixed-Q quadrature evaluation costs O(n log n + nKQ) arithmetic and is numerically, not algebraically, exact; no epsilon-approximation rate is certified. Each nonzero degree-K Horvitz-Thompson term has finite second moment exactly when n >= 2K; under the same assumptions the full surrogate gradient has finite second moment whenever n >= 2K (sharpness there is open). At K=1 the construction recovers classical priority sampling. All quantities require only the values and differentiable computation graphs of the n+1 drawn items' probabilities, so finite structured sequence policies sampled by exact SBS are covered. A certified finite-Q quadrature bound and countably infinite support remain open. Validation code is included as ancillary files.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。