在球面上构建流模型，高效生成离散序列数据。
Spherical Flows for Sampling Categorical Data
- 将离散序列建模置于球面空间，利用vMF分布设计噪声与得分函数。
- 通过降维求解速度方程，实现无需梯度的连续采样路径。
- 结合预测-校正采样，在数独、语言建模等任务上显著提升性能。
我们研究在连续嵌入空间中学习离散序列生成模型的问题。与以往在欧氏空间或概率单纯形上的方法不同，本文在球面 $\mathbb S^{d-1}$ 上进行建模。该空间下，冯·米塞斯-费舍尔（vMF）分布诱导出自然的噪声过程，并支持闭式条件得分。虽然条件速度一般不可解析，但利用vMF密度的径向对称性，我们将球面上的连续性方程简化为余弦相似度的标量常微分方程，其唯一有界解确定了速度。在 $(\mathbb S^{d-1})^L$ 上，边际速度与边际得分均可分解为后验加权的切向和，仅差于每标记的标量权重。这使得可同时实现基于常微分方程（ODE）与预测-校正（PC）的采样。后验为唯一需学习的对象，通过交叉熵损失训练。实验对比了vMF路径与测地线及欧氏路径的效果，结果表明，尤其是在结合PC采样时，该方法在数独、语言建模与数学推理任务上均显著优于基线。
原文摘要 · Abstract (English)
We study the problem of learning generative models for discrete sequences in a continuous embedding space. Whereas prior approaches typically operate in Euclidean space or on the probability simplex, we instead work on the sphere $\mathbb S^{d-1}$. There the von Mises-Fisher (vMF) distribution induces a natural noise process and admits a closed-form conditional score. The conditional velocity is in general intractable. Exploiting the radial symmetry of the vMF density we reduce the continuity equation on $\mathbb S^{d-1}$ to a scalar ODE in the cosine similarity, whose unique bounded solution determines the velocity. The marginal velocity and marginal score on $(\mathbb S^{d-1})^L$ both decompose into posterior-weighted tangent sums that differ only by per-token scalar weights. This gives access to both ODE and predictor-corrector (PC) sampling. The posterior is the only learned object, trained by a cross-entropy loss. Experiments compare the vMF path against geodesic and Euclidean alternatives. The vMF path especially in combination with PC sampling significantly improves results on Sudoku, language modeling, and mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。