提出无偏采样方法,高效估算转导语言模型的词元概率。
Stochastic Estimation of Transduced Language Models
- 基于无放回重采样与逆包含概率加权,获得无偏估计。
- 在文本和DNA数据上,计算效率更高且方差更小。
- 适合需精确概率估计的长序列建模任务,如基因翻译分析。
转导语言模型(TLM)将预训练的源语言模型与有限状态转导器结合,生成目标字符串的语言模型。计算目标前缀的概率需对所有映射到该前缀开头的源字符串进行求和,而此类源串可能呈指数级或无限多。先前方法依赖源前缀概率的简化计算,并通过阈值剪枝的束搜索近似求和,但仅提供下界且误差未知。本文提出一种无放回重采样策略,对每个选中前缀按其包含概率的倒数加权,递归应用该修正可得到目标前缀概率的无偏估计,并能估计阈值剪枝丢失的概率质量。所提束搜索算法在保留前缀的基础上动态扩展并采样,随着累计概率增加逐步减少前缀数量,从而节省计算量且以概率1保证终止。在百科文本和DNA序列上的实验表明,相比带放回的顺序蒙特卡洛基线,本方法在文本上实现更优的计算-方差权衡,在DNA上于相同粒子数时误差更低。在DNA到氨基酸的转导任务中,相比阈值剪枝束搜索,运行时间降低多个数量级,使长目标序列的前缀概率估计成为可能。将发表的阅读时间分析中的阈值剪枝替换为无偏采样后,语料库困惑度显著降低,但结论保持不变。
原文摘要 · Abstract (English)
Transduced language models (TLMs) compose a pretrained \emph{source} language model with a functional finite-state transducer to induce a language model over \emph{target} strings. Computing the probability of a target prefix under a TLM amounts to summing the source-model probabilities of all source strings that the transducer maps to target strings beginning with that prefix. This set can be exponentially large or infinite. Prior work uses a computational shortcut based on source prefix probabilities, then approximates the resulting sum with threshold-pruned beam summing. This produces a lower bound with unknown error. Instead, we resample source prefixes without replacement and reweight each selected prefix by the inverse of its inclusion probability. We show that applying this correction recursively gives an unbiased estimator of the target prefix probability and lets us estimate the mass lost by threshold pruning. Our beam-summing algorithm extends the retained source prefixes and samples which prefixes to keep, reducing their number as more probability mass is added to the running estimate. This can save computation and guarantees that the run halts with probability one. We evaluate the method on encyclopedic text and DNA against sequential Monte Carlo baselines that resample with replacement. It achieves a better compute--variance tradeoff on text and lower error at the same maximum number of particles on DNA. On a DNA-to-amino-acid transduction, it reduces runtime by several orders of magnitude relative to threshold-pruned beam summing and makes estimating prefix probabilities for long target strings feasible. Replacing threshold pruning with unbiased sampling in a published reading-time analysis substantially lowers the estimated corpus surprisal but leaves the published conclusions unchanged.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。