Bad science, new chapters

Jordi Varela
Editor

In July of last year I commented on a work by Paul Glasziou and Iain Chalmers that concluded that taking into account methodological shortcomings, unpublished studies and poorly explained ones, the waste of biomedical research could be of the order of 85%. According to these authors, only 15% of what is investigated reaches the clinic in conditions of quality and appropriate communication. A little over a year later, other sources insist that biomedical research, in general terms, does not show signs of recovery and, to make it understandable, I will follow the same scheme that I used in the post I have quoted.

Lack of rigour of many systematic reviews and meta-analyses

Systematic reviews and meta-analyses are two methodologies that accumulate, synthesize and evaluate the evidence from numerous individual studies, to resolve uncertainties, reduce biases, and delivering scientific information in a practical way to the busy universe of the clinic. There is data showing that something is happening with the rigour of these synthetic methodologies and the first concern is that of their strange hatchinge. John Ioannidis, in 2016, observed that the annual growth of reviews and meta-analyses was 150% since 1991, an increase in the production of epidemic characteristics, according to the author. This trend, according to Joshua Niforatos and Joshua Wallach (2019), has led to an editorial imbalance, since now practically one review is published for each original clinical trial.

The excess of systematic reviews and meta-analyses worries many researchers and clinicians. We now have reasonable doubts about the purpose, quality and credibility of many of these studies, in addition to promoting overlaps and contradictions between the different reviews that are committed to analyzing evidence from various angles on the same topic. In his article, Ioannidis offers some examples of overlaps, such as the 11 contradictory meta-analyses that have studied statin prevention of atrial fibrillation after cardiac surgery or the labyrinthine universe of meta-analyses of the effectiveness of antidepressants.

It is evident, says Wallach, that in the excessive growth of reviews and meta-analyses there are not only hidden interests, and there are also other circumstances that favour them. Many researchers prefer them, due to today's ease of managing databases, lower costs and controls (compared to clinical trials) and the greater number of citations enjoyed by reviews and meta-analyses, which increases the prestige of researchers, institutions and journals.

According to Wallach and Ioannidis, despite the misuse of the methodology and the poor quality observed, we have no alternative. In other words, systematic reviews and meta-analyses continue to be first-rate instruments to offer important scientific information on the existing evidence on a certain topic. But it would take a bit of rigour, to avoid damaging such a precious asset.

Waste of a lot of research that is not published

In the post from July 2019, I commented that half of the clinical trials funded by NIH were not published. But now I want to highlight the point of view of Harlan Krumholz, when he puts himself in the shoes of the 87,883 people, from 67 different studies, who accepted a clinical trial protocol, engaged with researchers, underwent the double-blind methodology, received tests and treatments that they did not know if they were real or not and, later, without any explanation, the researchers themselves or the funders, or whoever, decided that the results would not be published, and the question is: it would not be a duty researchers' morals that when they invite a person to enter a trial, they, for their part, commit to making the results public, even the negative ones? This loss of scientific information therefore also has a human aspect. Shouldn't responsibilities be asked? Krumholz asks (and I add myself).

Fake News in the scientific literature

In observation of 94 articles on the evaluation of new drugs in cancer, Matthew Abola and Vinay Prasad found that, in half of them, the editors had used superlative adjectives of the type: miraculous, revolutionary, innovative, wonderful or transformative: an obvious sign that something was not quite right. In this sense, the Lown Institute, on its blog, has published an analysis of fake news (reversal of a negative test to a positive) in cardiovascular research. Lown states that researchers from the CABANA project (2018), which measured the effectiveness of cardiac ablation, managed to reverse (and publish) a negative result into a positive one, and the question is how often do researchers present negative results as if they were positive? To respond, a team led by Muhammad Shahzeb Khan reviewed one hundred publications in high-impact journals of cardiovascular clinical trials and found that among the studies that had negative results, 67% had reversals in the main text, 52% in the abstract and 10% in the title.

I think that journals have a great responsibility not to act as a speaker for scientific fake news and, therefore, to detect reversals and pay attention that the titles, abstracts and texts reflect, faithfully, what the numbers say. We should remember that some studies with negative results, such as the ORBITA project, which demonstrated the ineffectiveness of stents in people with stable angina, can be very valuable for clinical practice.

In this new instalment of the "Bad Science" series, I once again insist on three principles: a) that research, reviews and meta-analysis must be of quality, transparent and collaborative, b) that everything that is investigated must be published, and c) that it must be published well, that after all, a negative result can also be very enlightening for clinicians who have to recommend and share decisions every day.

Jordi Varela
Editor