让离散语言模型学会理解词元间的距离关系，提升多领域性能。
Teaching Metric Distance to Discrete Autoregressive Language Models
- 用预定义词元距离构建奖励加权分布替代独热编码
- 在视觉定位、机器人操作等任务中显著提升数据效率和效果
- 无需采样与策略梯度，兼容现有训练流程，适合各类离散生成任务
大型语言模型以离散词元词汇进行自回归预测，已广泛应用于视觉、机器人和多模态推理等领域。然而，使用独热目标训练忽略了词元间的度量关系，限制了在数值、空间坐标或量化嵌入等需距离意义的任务表现。本文提出DIST2Loss，一种面向离散自回归模型的距离感知目标函数，将独热目标替换为基于预定义词元距离的奖励加权分布。该方法可被解释为具有已知词元奖励的熵正则化策略优化的闭式解，保留强化学习核心机制的同时避免采样、轨迹回放与不稳定性。实验表明，DIST2Loss在多个领域均提升数据效率与下游性能：在视觉定位中生成更紧致的边界框，加速机器人操作中的动作学习，改善大模型对齐的奖励建模，并增强向量量化图像生成能力。结果证明，距离感知监督为离散自回归模型提供了一种简单且通用的替代独热监督的方案。
原文摘要 · Abstract (English)
Large language models (LLMs) operate as autoregressive predictors over discrete token vocabularies, a formulation that has enabled their adaptation far beyond natural language to vision, robotics, and multimodal reasoning. However, training against one-hot targets disregards metric relationships between tokens and limits effectiveness on tasks where distance is meaningful, such as numerical values, spatial coordinates, or quantized embeddings. We introduce DIST2Loss, a distance-aware objective for discrete autoregressive models that replaces one-hot targets with reward-weighted distributions derived from predefined token distances. DIST2Loss can be interpreted as the closed-form solution to entropy-regularized policy optimization with known per-token rewards, retaining the core mechanism of reinforcement learning while avoiding sampling, rollouts, and instability. Our experiments show that DIST2Loss improves data efficiency and downstream performance across diverse domains. It yields tighter bounding boxes in visual grounding, accelerates robotic manipulation by improving action learning, enhances reward modeling for LLM alignment, and strengthens vector-quantized image generation. These results demonstrate that distance-aware supervision offers a simple and general alternative to one-hot supervision for discrete autoregressive models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。