A day-old chatbot exposed a gap Microsoft says it missed

The design

Microsoft's Technology and Research and Bing teams built Tay, described on the product's own tay.ai site, archived the day before its shutdown, as 'an artificial intelligent chat bot... to experiment with and conduct research on conversational understanding,' 'targeted at 18 to 24 year old in the US' and built to run on Kik, GroupMe and Twitter. The page states the bot got 'smarter' the more a person chatted with it, and discloses a data practice: conversations 'are anonymized and may be retained for up to one year to help improve the service,' alongside a tracked profile of 'nickname, gender, favorite food, zipcode, relationship status.' Tay followed Microsoft's Chinese chatbot Xiaoice; Microsoft's own account of the incident, published two days later, frames the connection directly, asking whether 'an AI like this' would be 'just as captivating in a radically different cultural environment.'

What the evidence says

Microsoft's own postmortem, a blog post published by Peter Lee on 25 March 2016, states 'in the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay,' after which 'Tay tweeted wildly inappropriate and reprehensible words and images,' leading Microsoft to take it offline. Lee calls this 'a critical oversight' regarding what the vulnerability allowed, attributing the failure to the system's design rather than any individual user's motive, which this entry follows: the post documents a company's account of its own software, not a forensic identification of who acted or why.

What it asks of people

Tay's design asked users for casual conversational input in exchange for a bot that adapted its replies, with the tay.ai page framing this as reciprocal: 'the more you chat with Tay the smarter she gets.' That same openness to unfiltered public input, Microsoft's postmortem states, is what the coordinated attack exploited, turning a feature the product was built to advertise into the mechanism of its failure within a day.

Privacy and safeguards

Tay's own site disclosed, before the incident, a retention period of up to one year and a route to deletion via the product's contact form, with a profile schema limited to five voluntary fields. Lee's postmortem adds a safeguard commitment rather than a data policy: Microsoft says it would address 'the specific vulnerability' before any similar release, recorded here as stated intent, not a verified subsequent audit.

• Does the product's own materials disclose how it changes based on what users tell it, and who can exploit that mechanism?
• When a company describes an incident as a 'vulnerability,' does its account distinguish a design failure from a moderation failure?
• What retention period and deletion process does the product's own privacy page actually specify?

Microsoft's own account of Tay is candid about the company's responsibility and specific about the mechanism, a learning system exposed to unfiltered public input, without naming or blaming particular users, which this entry follows. The episode is cited by name in the design of Microsoft's next chatbot, Zo, a connection documented in this archive's companion entry on that product's more cautious data practices.