arXiv:2411.04798cs.HCcs.IR2024-11被引 1

提出以目标为中心的排序模型设计框架,帮助团队平衡多目标冲突。

Orbit: A Framework for Designing and Evaluating Multi-objective Rankers

  • 以目标为核心构建排序模型设计流程,作为跨团队沟通桥梁
  • 通过交互系统实现实时探索多目标权衡,提升决策效率
  • 适合需要平衡用户参与、满意度等多重目标的推荐系统研发

生产环境中的机器学习需平衡多个目标,尤其在排序或推荐系统中,用户参与度、满意度、多样性与新颖性等目标常相互冲突。然而,多目标排序器的设计本质上是动态且复杂的问题——不存在单一最优解,需求也随时间演变。有效设计依赖跨职能团队协作及对大量信息的细致分析。本文提出Orbit,一种以目标为中心的排序器构建与迭代概念框架。该框架将目标置于设计中心,作为沟通边界对象,指导实践者进行设计与评估。我们实现了一个交互式系统,使利益相关方可直接探索目标空间,支持实时权衡分析。通过包含十二名行业实践者的用户研究验证,Orbit能高效探索设计空间,促进更明智决策,并增强对多目标内在权衡的认知。Orbit(1)为多目标机器学习模型开启目标导向设计新路径;(2)引导从业者突破狭隘的指标或样本中心思维。

原文摘要 · Abstract (English)

Machine learning in production needs to balance multiple objectives: This is particularly evident in ranking or recommendation models, where conflicting objectives such as user engagement, satisfaction, diversity, and novelty must be considered at the same time. However, designing multi-objective rankers is inherently a dynamic wicked problem -- there is no single optimal solution, and the needs evolve over time. Effective design requires collaboration between cross-functional teams and careful analysis of a wide range of information. In this work, we introduce Orbit, a conceptual framework for Objective-centric Ranker Building and Iteration. The framework places objectives at the center of the design process, to serve as boundary objects for communication and guide practitioners for design and evaluation. We implement Orbit as an interactive system, which enables stakeholders to interact with objective spaces directly and supports real-time exploration and evaluation of design trade-offs. We evaluate Orbit through a user study involving twelve industry practitioners, showing that it supports efficient design space exploration, leads to more informed decision-making, and enhances awareness of the inherent trade-offs of multiple objectives. Orbit (1) opens up new opportunities of an objective-centric design process for any multi-objective ML models, as well as (2) sheds light on future designs that push practitioners to go beyond a narrow metric-centric or example-centric mindset.

多目标优化推荐系统人机交互设计框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。