Psychological studies were found to be false in more than 50% of cases

2024 Author: Seth Attwood | [email protected]. Last modified: 2023-12-16 15:55

There are "strong postures" that build confidence and reduce stress hormones. When people hold a cup of warm drink in their hands, they become friendlier to those around them. Willpower is a resource that we spend when we resist temptation. The ability to postpone reward determines the child's future success.

These statements are very similar to each other: behind them are well-known psychological research, popular science bestsellers, columns in popular magazines and TED talks.

They also have one more thing in common: they all turned out to be wrong.

The reproducibility crisis has cast doubt on entire fields of science. Many of the results, which were widely quoted in the media, are now considered exaggerated or false. When scientists tried to replicate both classic and recent psychological experiments, the results were surprisingly consistent, with about half of the cases being successful and the other half being failing.

The crisis finally became apparent in 2015, when scientists led by Brian Nosek checked 100 psychological studies. They were able to achieve initial results only in 36 cases. The Lancet editor-in-chief Richard Horton soon stated:

“The accusations against science are quite straightforward: at least half of the scientific literature is simply wrong. Suffering from studies with a small sample size, scanty effect and incorrect analyzes, as well as an obsession with fashion trends of dubious importance, science has taken a turn towards ignorance.

Reproducibility is one of the key requirements for scientific knowledge. The better the result is reproduced, the more reliable it is - this is the only way to separate real patterns from simple coincidences

But it turned out that this requirement is not always met.

The crisis began with medicine, but most affected psychology. In the summer of 2018, scientists attempted to replicate a selection of psychological studies published in Science and Nature, the world's most prestigious scientific journals. Out of 21 experiments, only 13 were confirmed - and even in these cases, the original results were exaggerated by about 50%.

Often, the reproducibility test is failed by those studies that were widely replicated in the media and managed to influence public consciousness. For example, the works that search engines impair memory, and reading fiction develops the ability to empathize. If repeated experiments fail, this does not mean that the original hypotheses are worthless. But better research is now needed to prove them.

How to predict the future with statistics

In 2011, the famous American psychologist Daryl Boehm published an article that proved the possibility of clairvoyance. This conclusion was not a product of his violent imagination, but was based on decades of research, which involved hundreds of people. Many suspected that Boehm decided to arrange something like Sokal's scam and expose psychology with a fake article with deliberately absurd conclusions. But by all methodological standards, the article was very convincing.

In one of Behm's experiments, two screens were placed in front of the participants - they had to guess which image was hidden behind. The picture was generated randomly immediately after the selection was made. If the participants did a good job, it would indicate that they can somehow anticipate the future. The experiment used two types of images: neutral and pornographic.

Boehm suggested that if the sixth sense exists, it probably has an ancient evolutionary origin. If so, then it is more likely that it is attuned to our most ancient needs and urges

Participants guessed pornographic images 53% of the time - slightly more often than they should if they were pure chance. Given the large number of experiments, Boehm could claim that foresight exists.

Later, experts discovered that when analyzing the results, he used not entirely correct methods. As a rule, a research result is considered reliable if the probability that it was obtained by accident does not exceed 5%. But there are many ways to reduce this value to the required level: change the initial parameters of the analysis, add or remove the required number of examples to the sample, use more successful hypotheses after collecting the data.

The problem is that not only Boehm, but also many other scientists used the same techniques. According to a 2011 survey, almost half of psychologists admitted this

When the clairvoyant article came out, social scientists Joseph Simmons, Leaf Nelson, and Uri Simonson realized that science was heading for its own ruin. They built several computer models and found that using fairly standard statistical techniques, you can increase the level of false-positive results several times. This means that methods that are formally scientific can easily lead to completely absurd conclusions.

To illustrate this, scientists conducted an experiment that confirmed that listening to the song "When I'm Sixty-four" makes the listener a year and a half younger.

“Everyone knew that it was wrong to use such techniques, but they thought that this was a violation in its significance - like crossing the road in the wrong place. It turned out to be more like a bank robbery,”concluded Simmons.

How to tell bad research from good

It soon became clear that reproducibility issues were not limited to psychology. In cancer research, scientific evidence is supported in 10-25% of cases. In economics, 7 out of 18 laboratory experiments were unable to replicate. Artificial intelligence research also shows signs of crisis.

But losing faith in science, it seems, is still not worth it. Scientists have already come up with several ways that have greatly improved the reliability and quality of new research

Several years ago, almost no one published the results of repeated experiments, even if they were carried out. This was not accepted, did not bring grants and did not contribute to a successful scientific career. According to a Nature survey, more than 70% of psychologists have tried and failed to reproduce other people's research, about half have not been able to repeat their own, and almost no one has sought to publicize these results.

When the crisis of reproducibility surfaced, a lot has changed. Repeated research gradually became commonplace; experimental data began to be published more and more often in the public domain; journals began to publish negative results and record the overall plan of research even before they began.

Research has become more extensive - a sample of 30–40 people, which was quite standard in psychology, now suits very few people. Large international organizations - such as the Psychological Science Accelerator - are testing the same hypotheses in several laboratories around the world.

Before checking the articles from Nature and Science, which we wrote about at the beginning, scientists were asked to place bets on the sweepstakes. They had to predict which research would pass the test and which would fail. Overall, the rates were very accurate. “This means, firstly, that the scientific community can predict which works will be able to be repeated, and, secondly, that the impossibility of replicating the study was not a mere coincidence,” the organizers of the experiment say.

Scientists are generally good at distinguishing reliable from unreliable research - that's good news. Now experts from the Center for Open Science, together with the DARPA agency, are trying to create an algorithm that will perform the same task without human intervention.

There are too many articles published each year to manually recheck even a small fraction of them. If artificial intelligence gets down to business, everything will be much easier.

Already in the first tests, the AI successfully coped with predictions in 80% of cases.

What makes research unreliable most often? Small samples, inconsistencies in numbers, too beautiful confirmation of hypotheses. And also - the desire for sensations and too simple answers to difficult questions.

Too good to be true

The easiest way to create sensational research is through deception. The famous social psychologist Diederik Stapel used fabricated data in several dozen scientific articles. Stapel's research was spreading through newspapers and magazines at a tremendous speed, he received several prestigious scientific awards, was published in Science and was considered one of the largest specialists in his field.

Once it turned out that for a long time Stapel did not conduct research at all, but simply invented data and gave it to students for analysis.

This is very rare in science. Much more often loud, but incorrect statements arise for other reasons. People are desperately looking for simple, understandable and effective answers to exciting questions. It can be very easy to be tempted to think that you have these answers, even if you really don't. The pursuit of simplicity and certainty is one of the main reasons why many studies fail to test for reproducibility. Here are some notable examples.

Marshmallow experiment

In an experiment, children were asked to choose between one small reward - like a marshmallow - that could be received immediately, or a double reward if they could wait a bit. It later turned out that children who received the second award became more successful in adulthood. The study became very popular and influenced some school curricula.

In 2018, the experiment was repeated on a wider sample. It turned out that wealth in the family is a much more important factor, on which the level of self-control also depends.

"Poses of Strength" and "Poses of Weakness"

The participants in the experiment took one of two poses for two minutes: they leaned back in a chair and threw their legs on the table ("strength pose") or crossed their arms over their chest ("weakness pose"). As a result, the participants from the first group felt more confident and more often agreed to take risks in gambling. Those who sat in a strong position increased their testosterone levels, and those who sat in a weak position increased cortisol. In repeated experiments, only one effect was reproduced: "strength postures" helped the participants feel more confident, but did not change their behavior or hormonal parameters.

Associations with old age make you move more slowly

Participants in the experiment were asked to solve several puzzles. If words were inserted into them that are associated with old age - "forgetful", "elderly", "lonely" - then the participants left the room at a slower pace.

In recent trials, the experiment was successfully reproduced only in one case: if the experimenters themselves knew that in the tests the participants were hinting at old age. The effect itself remained, but the reasons were already different.

Warm objects make people friendlier

Participants in the experiment were allowed to hold a cup of hot or cold coffee for a short time, and then asked to rate the person's personality using a short description. Participants who held a hot cup of coffee rated the person as more likable. In another experiment, participants were given an item in a warm or cold package and then asked to keep it or give it to a friend. If the item was wrapped in a warm package, participants were more likely to choose the second option. Repeated experiments with a wider sample did not produce such results. It looks like warm clothes won't make you an altruist.

Willpower is depleted when we resist temptations

In front of the participants in the experiment were placed two plates - with cookies and radishes. In the first group, the participants were allowed to eat cookies, and in the second, only radishes. Then each participant was asked to solve an impossible puzzle. Participants who ate only radishes in the first part of the experiment gave up much earlier than others. In repeated experiments, the results were not confirmed.

In some cases, the capacity for self-control did not deplete, but even intensified over time. Many psychologists now consider the concept of "willpower" too simplistic.

Much has already been done in world psychology to make research more reliable and reproducible. In Russia, this problem has yet to be grasped.

“In Russian psychology, the problems of the crisis are mainly concerned with scientific youth, who are largely oriented toward Western science,” Ivan Ivanchey, associate professor at RANEPA, told the Knife. - Control over the quality of publications in Russian is generally not very high. The journals rarely reject articles, so a lot of low-quality research is published. Small samples are often used, which also reduces the likelihood of successful reproduction. There is a suspicion that if you seriously tackle the issue of reproducibility of Russian-language works, you will find many problems. But no one is directly involved in this."

In January 2019, it became known that the Russian government is going to significantly expand the requirements for scientists in terms of the number of publications: the minimum number of articles published per year should grow by 30-50%.

Scientists from the influential academic "July 1 Club" criticized the initiative: "The task of science is not to produce the maximum number of publications, but to explore the universe and benefit from the knowledge gained for humanity." Most likely, the new requirements will only increase the scale of the problem.

The story about the crisis of reproducibility is not a story about the coming apocalypse and the invasion of the barbarians. If the crisis had not occurred, everything would have been much worse: we would still refer to erroneous research in full confidence that we know the truth. Perhaps the time of bold headlines like "British scientists have proved" is coming to an end. But rumors that science is dead should be considered somewhat exaggerated.