arXiv:2511.03827cs.CL2025-11被引 1

通过固定间隔验证提升大模型对齐可靠性,降低计算成本。

STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models

  • 固定周期强制验证,摆脱对置信度的依赖
  • 在HH-RLHF上实现与顶尖方法相当的对齐效果
  • 适合需要高吞吐和稳定性的大规模部署场景

将大语言模型(LLM)对齐人类价值观对安全部署至关重要。推理时技术可实现生成过程的细粒度控制,但依赖模型内部的不确定性估计来分割输出。我们发现这带来两大缺陷:(a) 容易受高自信幻觉误导;(b) 因异步、稀疏批处理导致硬件利用率低。这两点共同降低对齐可靠性并增加令牌与计算开销,限制实际可扩展性。为此,基于动态推理对齐方法,我们提出STARS——同步令牌对齐鲁棒监督算法,一种解码时算法,通过在固定时间窗内强制验证来引导生成。该方法解耦分割与置信度,实现锁步并行执行,能可靠检测出不确定性指标遗漏的错误。在HH-RLHF基准测试中,STARS达到与最先进动态方法相当的对齐质量,同时严格控制拒绝成本并最大化系统吞吐量。此外,其性能显著优于微调及多种前沿推理解码策略,确立了固定周期采样作为大规模模型对齐的鲁棒、高效替代方案。代码已公开于https://github.com/purseclab/STARS。

原文摘要 · Abstract (English)

Aligning large language models (LLMs) with human values is crucial for safe deployment. Inference-time techniques offer granular control over generation; however, they rely on model uncertainty, meaning an internal estimate of how likely the model believes its next tokens or outputs are correct, for segmentation. We show that this introduces two critical limitations: (a) vulnerability to miscalibrated confident hallucinations and (b) poor hardware utilization due to asynchronous, ragged batch processing. Together, these issues reduce alignment reliability while increasing token and compute costs, which limits their practical scalability. To address these limitations, building on dynamic inference-time alignment methods, we introduce STARS, Synchronous Token Alignment for Robust Supervision, a decoding-time algorithm, which steers generation by enforcing verification at fixed-horizon intervals. By decoupling segmentation from confidence, STARS enables lockstep parallel execution and robustly detects errors that uncertainty metrics miss. On the HH-RLHF benchmark, we demonstrate that STARS achieves competitive alignment quality with that of state-of-the-art dynamic methods, while strictly bounding rejection costs and maximizing system throughput. Furthermore, it outperforms fine-tuning and several state-of-the-art inference-time decoding strategies by good margins, and establishes fixed-horizon sampling as a robust, system-efficient alternative for aligning LLMs at scale. The code is publicly available at https://github.com/purseclab/STARS.

大模型对齐推理优化批量处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。