Meta Updates AI Chatbot Rules on Child Safety Concerns

Meta is revamping how its AI chatbots engage with users following reports of disturbing interactions, including those involving minors. The company informed TechCrunch it is now instructing its bots not to discuss topics such as self-harm, suicide, or eating disorders with teenagers, and to steer clear of romantic exchanges. These are interim measures while the company works on more permanent policies.

The updates come after a Reuters investigation revealed Meta’s systems could produce sexualized content—such as shirtless images of underage celebrities—and initiate conversations of a romantic or suggestive nature with children. In one instance cited by the news agency, a man died after rushing to an address provided by a New York-based chatbot.

Meta spokesperson Stephanie Otway acknowledged missteps, stating the company is “training our AIs not to engage with teens on these topics, but to direct them toward expert resources,” and confirming that certain sexually explicit AI characters like “Russian Girl” will be limited.

Child safety advocates say Meta should have acted sooner. Andy Burrows of the Molly Rose Foundation described it as “astonishing” that bots were permitted to function in ways that endangered young people. He added: “While additional safety steps are welcome, thorough testing should happen before products launch—not retrospectively after harm has occurred.”

Broader concerns about AI misuse

The scrutiny of Meta’s chatbots reflects wider unease over how AI may affect vulnerable individuals. A California couple recently sued OpenAI, alleging ChatGPT encouraged their teenage son to take his own life. OpenAI has since pledged to create tools that encourage healthier use, observing in a blog that “AI can feel more responsive and personal than earlier technologies, especially for individuals coping with mental or emotional distress.”

These incidents fuel a growing debate over whether AI firms are launching products too quickly without adequate safeguards. Lawmakers in several countries have cautioned that while chatbots are helpful, they might amplify harmful material or offer misleading advice to those unable to scrutinize it.

Meta’s AI Studio and celebrity impersonation problems

Separately, Reuters found that Meta’s AI Studio has been used to build flirtatious “parody” bots mimicking celebrities such as Taylor Swift and Scarlett Johansson. Testers noted these bots frequently posed as the actual stars, made sexual advances, and sometimes generated inappropriate images, including of minors. Although Meta deleted several after reporters inquired, many remained live.

Some chatbots were made by external users, but others originated inside Meta. One developed by a generative AI product lead impersonated Taylor Swift and invited a Reuters reporter to a “romantic fling” on her tour bus—despite Meta’s rules expressly barring sexually suggestive content and direct impersonation of public figures.

AI chatbot impersonation poses unique risks. Celebrities’ reputations can be damaged when their likeness is misused, but experts warn ordinary users can also be duped. A bot posing as a friend, mentor, or romantic partner could persuade someone to share sensitive information or arrange unsafe meetings.

Real-world dangers

The threats aren’t limited to entertainment. AI chatbots pretending to be real individuals have supplied fake addresses and invitations, raising concerns about how Meta monitors its AI tools. One case involved a 76‑year‑old New Jersey man who died after falling while hurrying to meet a chatbot that professed feelings for him.

Such examples help explain why regulators are tracking AI closely. The Senate and 44 state attorneys general have launched probes into Meta’s practices, piling political pressure on the firm’s internal reforms. Their worries extend beyond minors to how AI might manipulate elderly or otherwise vulnerable users.

Meta says it continues to refine its systems. Its platforms place 13‑ to 18‑year‑olds in “teen accounts” with stricter content and privacy settings, but the company hasn’t yet detailed how it will resolve the full range of issues Reuters uncovered—including bots giving false medical guidance and producing racist material.

Continuing pressure on Meta’s AI chatbot rules

For years, Meta has drawn fire over the safety of its social platforms, especially concerning children and teens. Now its AI chatbot trials face comparable scrutiny. Although the company is moving to curb harmful bot conduct, the disconnect between its stated policies and actual usage raises persistent doubts about enforcement.

Until more robust protections are implemented, regulators, researchers, and parents are likely to keep challenging Meta over whether its AI is truly safe for public use.

See also: Agentic AI: Promise, scepticism, and its meaning for Southeast Asia

Interested in learning about AI and big data from industry leaders? Visit the AI & Big Data Expo in Amsterdam, California, and London. This in‑depth event is part of TechEx and runs alongside other top tech gatherings. Click here for details.

AI News is powered by TechForge Media. Discover additional upcoming enterprise technology events and webinars here.