Stanford's AI Index flags a privacy gap in romantic chatbots

The design

Stanford University's Human-Centered AI Institute (HAI) publishes an annual AI Index report; its 2024 edition, in the 'Responsible AI' chapter's notable-incidents section (Figure 3.1.4), highlights 'Privacy concerns with romantic AI chatbots,' describing such products as 'meant to resemble a lover or friend... and to be a companion for their users.' The Index text summarises research the Mozilla Foundation conducted separately: a review of 11 romantic AI chatbots for privacy risk, published by Mozilla's *Privacy Not Included project as 'Romantic AI Chatbots Don't Have Your Privacy at Heart' by Jen Caltrider, Misha Rykov and Zoë MacDonald on 14 February 2024.

What the evidence says

Mozilla's own review found that 90% of the 11 apps failed its Minimum Security Standards, with only one, Genesia AI Friend & Partner, meeting them; 73% had not published information on how they manage security vulnerabilities and 64% had not published clear encryption information; 45% allowed weak passwords; 90% may share or sell users' personal data, with EVA AI Chat Bot & Soulmate the sole app Mozilla did not flag on that count; and about 54% did not let all users delete their data. Replika, made by Luka, Inc., was the one app old enough for Mozilla to assess on 'track record,' and it received Mozilla's 'bad track record' flag. The AI Index quotes Crushon.AI's own privacy policy stating it 'may collect extensive personal and even health-related information... like your sexual health information.'

What it asks of people

Someone choosing a romantic or companion chatbot is, on this evidence, typically asked to disclose sensitive material to a product 'meant to... listen attentively,' in the Index's own phrase, without most apps in the review demonstrating they secure that material to a baseline standard or guarantee its later deletion.

Privacy and safeguards

Mozilla's review recorded an average of 2,663 third-party trackers detected per minute of app use across the 11 apps, with Romantic AI alone responsible for 24,354 in one minute; most companies reviewed stated they could share user data with government or law enforcement without requiring a court order. Neither Mozilla nor the AI Index report claims to have independently tested whether any company's stated safeguards work as described beyond what each company's own policy disclosed.

• Does a single failed security standard across 90% of reviewed apps change how a reader should weigh a companion app's other design claims?
• What would it take for a romantic AI chatbot to earn the kind of clean privacy assessment EVA AI received on data sale alone?
• How should a returning or long-term user weigh a 'track record' flag like Replika's against a newer app with no track record to assess?

Citing this Mozilla research, Stanford's AI Index treats romantic AI chatbots' privacy failures as a notable, documented pattern across an emerging product category, not as an allegation against any single named company.