If the peer reviewer at one journal says no to a scientific study, the researchers will generally move on to another, less prestigious journal, and will keep going like that until they can get the study published. There are so many journals that everything gets published somewhere in the end, no matter of how poor quality.
The whole system of peer-review builds on trust. The guiding principle is the idea that bad studies will be caught out over the long term, because when other people try to replicate the results, they won’t be able to.
There are two big problems with this line of thinking. The first is that scientific studies are expensive, so they often don’t get replicated, especially if they are big studies of drugs. For the most part, no-one but the drug company itself has the cash resources to do a follow-up study to make sure that the results are reliable. And if the drug company has done one study which shows a good effect, it won’t want to risk doing a second study that might show a weaker effect.
The second problem is that follow-up studies aren’t exciting. Being first is cool, and generates lots of media attention. Being second is boring. No-one cares about the people who re-did a study and determined that the results actually held up to scrutiny.
– Sebastian Rushworth, writing about How to understand scientific studies (in health and medicine)
Relevant article in the BMJ:
https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-health-research-is-fraudulent-until-proved-otherwise/
Final paragraph:
Dare we do as the BMJ suggests and assume this about some Covid research?
The second problem is that follow-up studies aren’t exciting. Being first is cool, and generates lots of media attention. Being second is boring.
True, but that overlooks the matter of the grant. There’s more to science than being first and getting on the telly. Before jam, there’s bread and butter.
No-one cares about the people who re-did a study and determined that the results actually held up to scrutiny.
Well, at least in theory, the government should be interested in whether the results hold up if they pertain to public policy, because Ministers, and even civil servants, would – at least at the margin – prefer to preside over success than failure. And since governments deploy lots of research funds, there would seem to be a simple solution to the problem :
1. Reserve say 40% of government research funding for replication studies
2. Make sure the quango that hands out the replication study money is different from the main quango
3. If possible try to arrange that the Chairman of the Replication Quango had an affair in college with the Chairman of the Main Research Quango, which ended very badly.
All data should be subject to replication.
After all, Elgazzar 2020 was recently withdrawn after a close look was taken at the data, and it seems that the authors had simply invented a lot of the results.
As Mr Rushworth says, An adage to remember when it comes to meta-analyses is “garbage in, garbage out” – a meta-analysis is only as good as the studies it includes.
A couple of meta-analyses used Elgazzar 2020 (only published as a pre-print and never peer-reviewed), and the results of these are now in question. Some researchers have concluded that, by removing Elgazzar 2020, the meta-analyses do not show a statistically significant result.
Elgazzar 2020 showed that ivermectin was effective against covid-19. The meta-analyses were by the BIRD group and FLCCC.
Dare we do as the BMJ suggests and assume this about some ivermectin research?
In theory. However in practice, it is assumed most bad stuff will either not be revealed until after they have moved on, or failing that, will just be memory-holed with the media uninterested.
Wasn’t it once a practice that a scientific finding was not accepted by the larger community until it had been replicated at least once, by someone not associated with the initial paper?
I seem to remember, from my history books, tales of scientists racing to replicate results from experiments done in the search for the ‘luminiferous ether’, and eagerly publishing reports of their success or failure to do so.
What happened to that?
It may be that there isn’t much interest in replicating existing work, but science progresses by building on the work of others. If your work fails because it’s built of faulty foundations, that will become apparent.
What Lee Moore said, in spades.
I was just thinking this yesterday. UKRI does not fund replication studies because they are not sexy for it to report.
Most scientists before about 1960 would seek to swiftly replicate (or otherwise) the relevant research of their rivals in order to understand what was going on. The advent of research council funding has stopped that in the UK.
If modern scientists can be persuaded to do this then we might get some value from the many millions spent by government on research.
IIRC, the doors of the Royal Society had a latin inscription on them translating as: “Take no one’s word for it.” Those guys may have known a thing or two about science.
A lot of health research is more akin to engineering than science anyway. It possibly needs a different approach.
@MadRocketSci Caveat Emptor?
There is not much money it proving that existing medications can, if used early and in the right combination, deal with diseases – so there will be few studies in this area (because there is nothing to patent – the medications are long standing).
In theory government studies should not be hit by this problem – but, in practice, they are. Government-university studies are also hit by the “Public Health” ideology. In the past medicine was about an individual doctor seeking to cure an individual patient, but “Public Health” ideology is about various political concepts (a political agenda), one can trace this all the way back to Sir Edwin Chadwick in the early 19th century – a follower of Jeremy Bentham who had his Collectivist conclusions written out before his “studies” (in various fields) were even started.
The result of all this is that vast numbers of people die of diseases, such as Covid 19, when Early Treatment could have saved them.
Tony Fauci recently won a million Dollar Israeli prize for “speaking truth to power” – doubly ironic as Tony Fauci was the person with the power, and he is not in the habit of speaking the truth.
I once was landed with a – not exactly replication – job for my master’s thesis, bsck in the early Sixties. It was to build part of an apparatus for polarizing atomic beams. This involved both shaped magnetic fields, and RF electrical fields. My advisor handed me my predecessor’s thesis, and pointed to a graph of the magnetic field in his device. It took the usual form of a graph – a line, with dots on it. But those dots were supposed to represent measured values. The line was very straight, and the dots quite close.
“Did you actually measure all those points?” the advisor asked.
“Uh, no,” my predecessor said. “I thought the dots made the line look better.”
This probably ended his career as a scientist. I did the job my own way, and made damn sure all of the dots were measured. I had more apparatus made, and tested my apparatus. It worked.
I got my degree, the article passed peer review, was published, and I continued in grad school. The polarizer stayed in use until that particular linac was decommissioned. I guess the standards were different in the Sixties, just like Clovis mentions.
Isaac Newton:
Peer review of his alchemy: Great work.
Peer review of his physics: Bafflement.
Mr Ed – the modern Vice Chancellor of Cambridge would kick Newton out, because he used nasty language.
He would never get into a position to do any work there.
Oddly enough the Vice Chancellor of Oxford might tolerant a Newton – but the left in Oxford are working hard to get rid of her.
Still physics is a rather intolerant field – if Newton returned to this Earth and decided to defend his ideas against the attack of Einstein, he would not be given a fair hearing. At least I do not believe he would be given a fair hearing.
@Paul Marks
Mr Ed – the modern Vice Chancellor of Cambridge would kick Newton out, because he used nasty language.
It is worth remembering, as we wax nostalgic about the past, that Newton almost didn’t get a fellowship at Cambridge because he refused to take holy orders due to his unconventional views on the Trinity. Being in holy orders was, at that time, necessary to be a fellow. He only got in because the king gave him a special dispensation.
So, things were pretty crappy in the past too.
[…] – Paul Marks […]
S’funny how often Newton surfaces in our little world.
By example: In 1666, Isaac was in isolation (see Bubonic Plague lockdown), at just 23 years old. Probaby getting bored at having to be living at home with his parents again. Doh! So he started playing around with simple expressions like (1+x)^n as a Binomial Theorem.
The coefficients are just the numbers in Pascal’s Triangle. People already knew that, so what, what’s new? Our Isaac went one step further and played around with (1+x)^-n
That wasn’t enough, he then started playing around with n = 1/2. Cut a long story short, thanks to his new-fangled Calculus (Theory of Fluxions) it calculated Pi to a much much higher precision than had ever ever been achieved before in 2,000 years.
The point? Nobody (I mean nobody) would have been able to Peer Review that, because there was nobody who was a peer. Peers review consensus. Newton smashed the consensus.