Meta’s AI scandal shows the system is working as designed

Sejal Sharma is IE’s AI columnist, offering deep dives into the world of artificial intelligence and its transformative impact across industries. Her bi-monthly AI Logs column explores the latest trends, breakthroughs, and ethical dilemmas in AI, delivering expert analysis and fresh insights. To stay informed, subscribe to our AI Logs newsletter for exclusive content.

Meta’s leaked chatbot rules are not a glitch in the system. They are the system. What Reuters exposed last week—guidelines that allowed bots to flirt with children, sexualize minors, or generate racist screeds—wasn’t an “error” tucked into a 200-page document. It was a policy signed off by lawyers, engineers, ethicists, and executives. Meta now calls those passages “erroneous.” But the question isn’t how they slipped in; it’s why they existed in the first place.

We’ve seen this story before: tech firms push the boundaries of acceptable harm, wait for the outrage, then patch things over just enough to move on.

In 2023, a Florida teenager died by suicide after weeks of conversations with a chatbot on Character AI. That bot reportedly encouraged harmful behavior instead of defusing it. The story caused outrage, but the underlying issue remains unchanged. AI is being deployed at a mass scale on apps used by millions of young people, yet the guardrails are only fixed after tragedies, scandals, or exposures by the media.

The pattern isn’t new

If this sounds familiar, AI chatbots have a long history of behaving badly, even before OpenAI came out with ChatGPT in November 2022 and breathed new life into artificial intelligence.

In 2016, Microsoft released Tay, an experimental Twitter bot designed to mimic how young people speak online. Within 24 hours, Tay had absorbed the worst of the internet and began spewing racist slurs, denying the Holocaust, and praising Adolf Hitler. Not much different from what Grok did when it was launched. Microsoft quickly pulled the plug, apologizing for what it framed as an unforeseen disaster. But in reality, it was the logical outcome of releasing an untested system into an online ecosystem with no meaningful safeguards.

Fast forward a few years, and the same patterns repeat. In 2022, Replika, an AI companion app with millions of users, came under fire when chatbots engaged in sexually explicit roleplay with minors. Italian regulators eventually banned the app from interacting with children, citing the obvious risks. Yet even after the ban, reports surfaced of adults becoming emotionally dependent on their AI ‘partners,’ with the bots sometimes encouraging harmful or unhealthy behavior.

Then came Snapchat’s ‘My AI’ in 2023, introduced as a safe and fun companion for teenagers. Within weeks, journalists demonstrated that the bot could advise a 13-year-old on how to hide alcohol from parents or give sexual guidance to a child. Snapchat scrambled to tighten guardrails, but the damage to trust was done by then.

Safety always comes after harm

And, of course, we have to go back to the Florida case involving Character AI, because it was touted then as the first such death directly involving AI. According to the boy’s parents, the bot developed an unhealthy and manipulative dynamic with their son, allegedly even encouraging suicidal thoughts. Character AI has denied responsibility, but the tragic outcome speaks for itself. A vulnerable teenager was left to the mercy of a system never designed to recognize or respond responsibly to mental health crises.

And it’s not just limited to AI. Even before the big AI race started in 2022, it wasn’t like big tech’s track record on safety in general was great. Think of Cambridge Analytica, where Facebook’s loose data-sharing practices enabled massive political manipulation; or Google’s Street View program secretly collecting Wi-Fi data from households under the guise of mapping; or YouTube’s recommendation algorithm pushing children toward harmful and extremist content. 

Apple, too, has faced backlash for issues like child labor in its supply chain and controversial scanning practices for child sexual abuse material that raised major privacy concerns. Amazon has repeatedly been scrutinized for worker exploitation in its warehouses, as well as for Alexa recording conversations without consent. 

What ties these examples together is not just that tech misbehaved. It’s that the companies only acted once the behavior became public. AI firms consistently treat safety as a problem of public relations, not product design. Meta’s leaked guidelines make this clear. 

Blaming “hallucinations” hides real choices

In the case of AI, one reason companies get away with this is that these systems still carry an aura of unpredictability. It hallucinates, makes mistakes with confidence, and often produces results that are difficult to verify without human oversight. When a chatbot produces something shocking, executives describe it as an unfortunate ‘hallucination,’ a fluke of the algorithm. This framing shifts responsibility away from the company and onto the mysterious nature of the technology itself.

But the Meta document breaks that illusion. The company wasn’t just dealing with an unpredictable chatbot but deciding what harm was acceptable. This isn’t unpredictability, it’s policy. And that makes the issue even more urgent. When a company as powerful as Meta includes these choices in official guidelines, it signals negligence and a willingness to normalize harm until outside scrutiny forces a rethink.

The stakes are highest for young people

The stakes are highest for young people, who make up a large share of chatbot users. More teenagers in the US are using AI chatbots for advice, companionship, and emotional support. A study by Common Sense Media found that over 70% of teens have tried AI companions, and nearly half use them regularly. 

Teenagers are drawn to these systems for companionship, curiosity, and entertainment. For many, an AI bot feels safer than confiding in parents or peers, without fearing judgment. This makes the younger generation more susceptible and vulnerable to falling prey to the whims of a bot. The trend raises questions for educators and policymakers about whether schools should teach digital literacy and how prepared students are to handle these tools’ ethical and emotional challenges.

When a bot sexualizes a child, encourages secrecy, or glamorizes self-harm, the damage is not abstract; it is deeply personal. What makes this worse is that AI chatbots are being incorporated into social media platforms that struggle to protect young users. Instagram, WhatsApp, and Snapchat are central to teenage social life.

Embedding AI into those spaces without robust safeguards drops an experimental technology into the laps of millions of adolescents and asks them to be the stress testers.

Regulation is still toothless

Regulation has so far failed to keep pace. In most countries, no binding rules require companies to test AI systems for safety before releasing them to the public. The powerful big tech lobby shoots down any law remotely alludes to testing and regulating AI machines. Hello there, SB 1047. Instead, firms rely on voluntary standards, internal ethics teams, and after-the-fact moderation. 

European regulators have been more proactive. Italy’s temporary ban on Replika and the EU’s emerging AI Act signal an appetite for firmer rules. However, in the US and elsewhere, oversight remains limited to hearings, statements of concern, and non-binding guidelines. The burden of accountability still falls on journalists, researchers, and grieving families to surface harms.

This imbalance allows Big Tech to set the terms. Companies decide what risks are “acceptable,” quietly codify them, and only adjust when public outrage leaves them no choice.

Meta’s chatbot scandal should not be dismissed as a PR misstep or a bad paragraph buried in internal documents. It is evidence of a deeper truth: Big Tech treats safety as an afterthought, only ever tightening guardrails once the damage is done.

The danger now is that we’re outsourcing emotional labor, companionship, and even children’s wellbeing to systems whose “acceptable risks” were drawn up in boardrooms. What Meta normalized in private, millions of users are being asked to endure in public.