WHO flagged chatbot risk for suicidal users years before US advisories

The design

On 28 June 2021, the World Health Organization published Ethics and Governance of Artificial Intelligence for Health, a 165-page guidance document the WHO's own publication page describes as the product of eighteen months of deliberation among experts in ethics, digital technology, law, human rights, and health ministries. The report itself sets out six consensus principles for AI used in health, and devotes a section to what it calls the shift toward AI-driven "clinical" care delivered outside the formal health-care system, including text or speech chatbots whose use, the report notes, accelerated during the COVID-19 pandemic.

What the evidence says

The report is WHO guidance, not a binding regulation, and it says so on its own publication page. Its substantive text goes further than the general framing on the landing page: discussing AI applications reaching users outside clinical settings, the report states plainly that "patient safety may be compromised if individuals are not connected to health-care services, such as lack of assistance to individuals with suicidal ideation who use an AI chatbot," and separately warns that such applications' efficacy "may not have been tested properly." This is WHO's own assessment of a risk category, not a finding about any named companion or mental-health chatbot product, and predates the APA and US Surgeon General advisories this site already covers.

What it asks of people

The report addresses governments and health systems, asking them to identify AI applications operating outside formal health-care oversight, set common standards, and, where warranted, prevent some applications from being deployed publicly, while ensuring people retain access to services that cannot be provided online. It does not ask anything of an individual user beyond implicitly cautioning that a chatbot offering mental-health-adjacent support is not equivalent to a connection to a health-care service, a distinction the report draws for policymakers rather than as advice to any specific reader.

Privacy and safeguards

Among its concerns, the report states that applications built outside the health-care system "may not meet the standards of privacy required for sensitive health data," and that because such applications are not always labelled as health-care services, governments may not even be aware they exist. The report does not name or assess any specific companion app's privacy practices; its contribution is naming the structural gap between an unregulated conversational tool and the oversight a formal health-care service would carry, without asserting that gap has caused a specific documented harm.

• Does a given AI companion or wellness chatbot describe itself as a health-care service, or does it disclaim that role, and does its privacy policy meet health-data standards either way?
• Has any government cited this WHO report when regulating chatbots operating outside formal health-care oversight?
• Where a chatbot interacts with a user expressing distress, what connection, if any, does it offer to an actual health-care service?

Published three years before the site's more recent US-focused advisories, the WHO report shows the concern about AI chatbots and vulnerable users forming as a governance question well before any single companion-app controversy made headlines.