解决大模型推理时嵌入空间拥挤问题，提升生成质量与多样性。
Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning
- 基于嵌入空间几何关系重加权，缓解采样时的令牌聚集现象。
- 在数学推理任务中，减少拥挤可提升成功率12.3%以上。
- 无需训练、兼容现有方法，适合各类大模型推理场景。
基于采样的解码是大语言模型复杂推理的核心，其策略显著影响模型行为。温度和截断等方法通过全局概率重加权或阈值处理，在生成质量与多样性间取得平衡。然而，这些方法仅作用于词元概率，忽略了嵌入空间中词元间的细粒度关系。本文发现一种新现象：嵌入空间拥挤，即下一个词元分布的概率质量集中在嵌入空间中几何相近的词元上。我们在多粒度下量化了这一现象，并发现其与数学问题求解的成功率存在统计关联。受此启发，我们提出CraEG——一种无需训练、单次遍历的即插即用采样方法，通过几何引导的重加权缓解拥挤问题。在多个模型和基准上的实验表明，该方法显著提升了生成性能，尤其在鲁棒性和多样性指标上均有改善。
原文摘要 · Abstract (English)
Sampling-based decoding underlies complex reasoning in large language models (LLMs), where decoding strategies critically shape model behavior. Temperature- and truncation-based methods reshape the next-token distribution through global probability reweighting or thresholding to balance the quality-diversity tradeoff. However, they operate solely on token probabilities, ignoring fine-grained relationships among tokens in the embedding space. We uncover a novel phenomenon, embedding-space crowding, where the next-token distribution concentrates its probability mass on geometrically close tokens in the embedding space. We quantify crowding at multiple granularities and find a statistical association with reasoning success in mathematical problem solving. Motivated by this finding, we propose CraEG, a plug-and-play sampling method that mitigates crowding through geometry-guided reweighting. CraEG is training-free, single-pass, and compatible with standard sampling strategies. Experiments on multiple models and benchmarks demonstrate improved generation performance, with gains in robustness and diversity metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。