A 2019 paper defined chatbot success as turns per session

The design

In a paper titled 'The Design and Implementation of XiaoIce, an Empathetic Social Chatbot,' Microsoft researchers Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum describe XiaoIce, the social chatbot Microsoft launched in China in 2014, as built on what they call an 'empathetic computing framework' intended to let the system 'recognize human feelings and states, understand user intents and respond to user needs dynamically.' The paper, first posted in December 2018 and revised in September 2019, states that the authors optimize XiaoIce for long-term user engagement, which they measure using a metric they name Conversation-turns Per Session, or CPS, defined as 'the average number of conversation-turns between the chatbot and the user in a conversational session,' where a larger number indicates, in the authors' framing, better engagement. The same paper is also listed as a Microsoft technical report, dated December 2018.

What the evidence says

The paper is explicit that CPS, not task completion, is its chosen success measure, and it states a reason: the authors write that they evaluate CPS as an average across millions of active users over a long period of time, typically one to six months, specifically to avoid rewarding short-term tricks, noting that bland but interactive replies can inflate turn counts briefly while ultimately reducing long-term engagement. The authors report that 'XiaoIce has achieved an average CPS of 23,' which they describe as significantly higher than that of other chatbots and even human conversations, a figure and comparison stated by the paper's own authors, based on their own logs, and not independently replicated in any source located for this entry.

What it asks of people

A system designed around CPS, by the authors' own description, treats a longer conversation as a better outcome, distinct from a system designed to resolve a user's request efficiently. The paper documents a specific tension it identifies itself: incorporating task-completion skills often reduces CPS in the short term, even as the authors argue such skills build the trust that sustains engagement over time. That is the authors' own stated design tradeoff, not an external critique of it.

Privacy and safeguards

The paper is a technical and design description, not a privacy or safety document, and it does not address data retention, consent, or moderation practices for XiaoIce's logged conversations. It states XiaoIce 'has communicated with over 660 million active users' since its 2014 launch, a figure the authors attribute to their own internal logs. A reader should treat CPS as a design metric documented by its creators, not a safety or wellbeing measure, since the paper does not claim it is one.

• Does a chatbot's own published success metric reward sustained engagement, task completion, or both?
• Who verifies a company's self-reported engagement statistics, if anyone outside the company?
• Does optimizing for conversation length align with, or diverge from, a user's own goal for a given chat?

The XiaoIce paper is unusually candid about what it optimizes for and why; that candor does not substitute for independent verification of the reported CPS figures, which remain, on the present record, self-reported.