No, That Viral Study Doesn't Show You Can Improve Your Mental Health by Deactivating Instagram

A study by Stanford’s Institute for Economic Policy Research (SIEPR) on the effects of social media went viral on X over the weekend. While the post represents the results as “shocking,” the study itself found little evidence that social media use hurts its users.

The SIEPR study was published as a working paper in April with the National Bureau of Economic Research. Of the 27 co-authors, most of whom are associated with American universities, eight are researchers from Meta, the parent company of Instagram and Facebook. The researchers recruited 19,857 Facebook users and 15,585 Instagram users to carry out “the largest-ever experimental study on the effect of social media deactivation on users’ emotional state.”

More than a quarter of the Facebook and Instagram users were assigned to treatment groups and were paid to deactivate their respective accounts for six weeks leading up to the 2020 presidential election. (All other users were part of the control group, which required users to deactivate their accounts for only the first of the six weeks.) Researchers conducted surveys on self-reported happiness, depression, and anxiety before and after the experiment. These metrics were combined to make a joint “emotional state index” (ESI).

The X post emphasizes that users who deactivated Instagram enjoyed an improvement of about 0.04 standard deviations in their ESI while users who deactivated Facebook enjoyed an improvement of roughly 0.06 standard deviations. But the authors themselves reported that the effect of deactivating Instagram on ESI is statistically insignificant after adjusting for multiple hypothesis testing. Moreover, the effect of deactivating Instagram on anxiety and depression was statistically indistinguishable from zero.

In the second case, the effect of deactivating Facebook on anxiety was also indistinguishable from zero. However, the effects on depression and ESI were statistically significant. The authors contextualize their results by explaining that the average of the six effects—Facebook deactivation on happiness, anxiety, and depression and Instagram deactivation on happiness, anxiety, and depression—is 0.038 standard deviations, which is “equivalent to 3.8 percent of people saying they feel happy ‘often’ instead of ‘sometimes.’”

Statistical significance does not necessarily imply substantial real-world differences. In this case, it does not. Christopher Ferguson, a professor of psychology at Stetson University, says that the threshold for distinguishing real psychological effects from statistical noise is much higher (0.21 standard deviations) than what was measured in the study. The standard for clinical significance, which he defines as “an effect people might actually begin to notice in the real world,” is higher still (0.41 standard deviations). Ferguson cautions that “a high proportion of nonsense relationships become ‘statistically significant’ with large datasets” and that “false positives…shouldn’t be interpreted as hypothesis supportive.” Ferguson also says that while the survey questions appear direct, they “are not clinically validated measures of depression or anxiety.”

The study suffers from additional methodological constraints. The fact that the study focuses on a specific historical time period—the six weeks leading up to the 2020 U.S. presidential election—raises questions about external validity; it “tells us very little about day-to-day interactions on social media,” says Ferguson. The authors themselves urge caution about generalizing results outside their sample because “less than one percent of the people who were invited to the study completed the experiment.”

Ferguson says the study is being widely represented as “supporting the idea that reducing social media time improves mental health outcomes when…it found no reliable evidence for such a relationship.”

The post No, That Viral Study Doesn’t Show You Can Improve Your Mental Health by Deactivating Instagram appeared first on Reason.com.