In 2021, TikTok changed its algorithm to stop users being flooded with harmful content. But it did not give the safer version to everyone. It held 10% of US users on the old version, about 15 million people at the time. It wanted to test whether the safer feed was less addictive. That is according to a confidential document reported by Bloomberg.

The document says that control group included Chase Nasca, a 16-year-old from New York. His account was in the test. As a result, it says, “TikTok’s filter bubble prevention strategies did not take effect on this user by design”. His feed then served him a stream of suicide, self-harm and depressive content, the document says. He died by suicide in February 2022.

What the document says

The experiment was an A/B test, or “backtest”. It is a standard technique that runs two groups to measure the effect of a change. Here, Bloomberg reports, the change under test was the safety system itself. TikTok wanted to know whether the safer feed reduced “stickiness”, a metric Wall Street watches. The test ran for up to a year.

TikTok wrote the 14-page document in March 2023, and it is under a court seal, Bloomberg reports. It compiled the review only after Businessweek asked about teen safety. It reviewed the 7,563 videos Chase saw in his final two weeks. It found 73% carried themes such as sadness or mental-health struggles that did not break its rules. Almost 10% did break them.

The document made two recommendations. It said the control group should drop to under 1%, and the test should last seven days, not up to a year. Even then, it noted, a 1% group would still leave “at least 1.8 million US users” without the update. TikTok says the 2022 test was routine, and that it has since refined its testing.

Why “by design” matters

Those two words carry weight in court. Thousands of lawsuits argue that social platforms are dangerous by design, a claim the companies, including TikTok, deny. In March, a California jury found Google’s YouTube and Meta’s Instagram negligent for targeting children with addictive features. It awarded one plaintiff $6m. The Chase document surfaced during that wider litigation.

TikTok gave Bloomberg a statement. “Our hearts break for any family that experienced a tragic loss,” a spokesperson said. The company is “deeply committed to the safety and well-being of users”, especially teens, they added, and invests heavily in detection and enforcement. TikTok and Snap settled on the eve of the California trial. On Monday, TikTok finalised three more settlements.

What TikTok told Congress

Two weeks after the internal review, chief executive Shou Chew testified to Congress. “Safety and wellness, in particular for teens, is a core priority for TikTok,” he said. One representative, Gus Bilirakis, gestured to Chase’s parents in the gallery and said, “your company destroyed their lives”. Another, Gary Palmer, said that each time Chew said “transparency”, “what I’ve heard is deception”.

In written answers dated May 2023, Congress asked TikTok how many user suicides it knew of. The company replied that it “does not believe that use of the platform has caused users to commit suicide”. That answer, Bloomberg notes, came two months after TikTok reviewed Chase’s account. The written answers did not mention the backtest.

The wider picture

A former TikTok trust and safety team leader, there when the test ran, told Bloomberg he had not known about it. He said A/B tests are common in the industry, but that this one was flawed. The control group was too large, he said, and it should have excluded minors and content such as suicide videos. TikTok did not address those points directly.

Chase’s parents sued TikTok for wrongful death in 2023. A judge dismissed the case last year, ruling that Section 230 shielded the company. An appeal is pending. The case is part of a widening reckoning over young users and social media. Courts are weighing liability, and governments are moving to restrict access for minors.

This story discusses suicide. If you or someone you know is struggling, support is available. A global list of helplines is at befrienders.org, and in the US you can call or text 988.

Get the TNW newsletter

Get the most important tech news in your inbox each week.