The secret life of trials....

When trials disagree...it can get ugly! But going into meta-analysis could help sort things out.

For a meta-analysis - a technique for combining the results of multiple trials - trials have to pretty much belong together. Differences might be responsible for contradictory results - including differences in the people in the trials, the way they were treated, or the way the trials were done. That's called heterogeneity. Too much of it, and the trials shouldn't be together. But heterogeneity isn't always a deal breaker.

Want to read more about heterogeneity in systematic reviews? Here's an article by Paul Glasziou and Sharon Sanders from Statistics in Medicine [PDF]. Or try the open learning materials from the Cochrane Collaboration.

Promising = over-hyped + under-tested

We've apparently been using the word "promising" to mean "showing signs of future excellence" since about 1600. I first wrote about the tendency of "promising" treatments to metamorphose into "disappointing" treatments in a BMJ piece about evidence based mistakes. Early results, after all, can't promise anything at all.

For all sorts of reasons, research findings are themselves over-positive. That includes the most highly cited clinical studies in the "best" journals: in a study back in 2005, over 30% were judged to be contradicted or turned out to have over-estimated benefits; and in another in 2019, over 10% of randomized trials reversed previous findings.

My cartoon graph depicts a cumulative meta-analysis: each new study is being absorbed into a summation of the evidence so far. With 4 studies, it's shifted from the "this helps" side of the ledger over to the "this harms" side. See more about cumulative analyses in this classic article.

Speaking of classic articles, here's another: in the BMJ's 2015 tongue-in-cheek christmas issue, "promising" was one of the positive hype words Christiaan Vinkers and colleagues analyzed in PubMed's abstracts - from 1974 to 2014.

"Novel" was another favorite - it was one of the words with an increase of up to 15,000%: "At this rate, we predict that the word 'novel' will appear in every record by the year 2123"! A bold, innovative take! Novel was a major one in a study of rising hype in the abstracts of applications for NIH funding, too. In which the authors discuss the term "semantic bleaching": the overuse of hyperbole can bleach out a word's meaning (Millar 2022).

So is "promising" still increasing? Yep. It was in 2.4% of 2014's PubMed records with abstracts, 3.3% in 2016 – and 4.5% in 2021! (My calculations are included in this post.)

"Promising" is a media and marketing staple, too. In the last 24 hours, you can find "promising" results in Google News for kidney cancer "options", a skin antibiotic test, candidates for new antibiotics, leishmaniasis treatment, skin infections, a "novel" anti-tumor DNA vaccine, marijuana for Hepatitis C...

There used to be several initiatives worldwide aiming to keep the media to account on this, story by story. In 2022, though, I could only find a couple still going strong: Germany's Median-Doktor.de and Japan's Media Doctor.

The breathless hype marches relentlessly on. On the plus side, at least words like "promising" mean that sometimes at least, marketing or optimism bias comes clearly labeled!



This post was updated on 25 February 2017, adding the study and data on the use of "promising" in PubMed. It was updated again on 16 May 2020, adding the 2019 study on medical reversals. And again on 15 November 2022 updating the data on "promising" in PubMed, and adding the source for my calculations; adding the study of NIH abstracts; and deleting NHS' Behind the Headlines and the US Health News Review, which have been discontinued.

Begging hopefully for less bias

From my guest blog post at Scientific American - Holy sacred cow!

Personal bias, wishful thinking plus biased research results is one recipe for a sacred cow. If more rigorous research results in a conflicting message, it could cause cognitive dissonance - and the less biased research often faces an uphill battle for acceptance.

And I also wrote about the importance of being just as rigorous about the claims we want to believe as those we're skeptical of here at the British Journal of Sports Medicine blog.

If you want to get better at critically assessing health claims, Smart Health Choices is a great place to start.

In 2018, the Centre for Evidence-Based Medicine at Oxford University began a Catalogue of Bias - with a blog and Twitter.


[Updated on 11 March 2018]

The trial acronymania menace

As if there's not enough for us to remember, we're supposed to remember endless acronyms for trials too now. There's even a wiki to help us keep them straight and a call for a register of trial acronyms to reduce multiple use of all the words ending in T!

Somewhere along the line this became marketing: not much equipoise in ACHIEVE, MIRACLE, PROMISE, or IMPROVE-IT, eh? A study has classified this as a form of coercion. Ivan Oransky called for a HALT (Help Acronyms Leave Trials). If you're irritated by the next outbreak of trial acronymania or acronymesis you come across, you're not alone!


Another trial acronym here at Statistically Funny.....Meet the AGHAST Investigators!



(Updated with IMPROVE-IT on 16 July 2019.)

Why oh why not randomize?

The over-abundance of over-diagnosis

Finding and aggressively treating non-symptomatic disease that would never have made people sick, inventing new conditions and re-defining the thresholds for old ones: will there be anyone healthy left at all?

Cochrane reviews: coming soon!

From my contribution to the Cochrane Collaboration blog: Cochrane @ PubMed Health

If "Cochrane" and "CD005032" aren't currently part of your worldview, you can read more about the Cochrane Collaboration here.

"Evidence-based" is the new "natural"

Evidence-based should mean there is a systematic approach behind the work seeking to minimize bias and rely as much as possible on research that also minimizes bias. You can read more about the basic principles here.

Established, experienced...and wrong

I've heard versions of this "increasing confidence" aphorism for years, but recently wondered it came from. This seems to be its source - A Skeptic's Medical Dictionary, by Michael O'Donnell:

     Clinical experience. Making the same mistakes with increasing confidence over an impressive number of years.

     Evidence-based medicine. Perpetuating other people's mistakes instead of your own.

(Cited in The Lancet - and see the book review in The BMJ.)

Shrikant Kalegaonkar pointed out that Oscar Wilde said something similar - and it is as exquisite as you would expect. It's here, in his 1890 The Picture of Dorian Gray:

He began to wonder whether we could ever make psychology so absolute a science that each little spring of life would be revealed to us. As it was, we always misunderstood ourselves and rarely understood others. Experience was of no ethical value. It was merely the name men gave to their mistakes. Moralists had, as a rule, regarded it as a mode of warning, had claimed for it a certain ethical efficacy in the formation of character, had praised it as something that taught us what to follow and showed us what to avoid. But there was no motive power in experience. It was as little of an active cause as conscience itself. All that it really demonstrated was that our future would be the same as our past, and that the sin we had done once, and with loathing, we would do many times, and with joy.

It was clear to him that the experimental method was the only method by which one could arrive at any scientific analysis of the passions; and certainly Dorian Gray was a subject made to his hand, and seemed to promise rich and fruitful results. 

From the sublime to the ridiculous. I've another to add to this picture. It's based on an aphorism I coined myself in a piece I wrote in The BMJ in 2004 - cartoon version and post here on Statistically Funny:

     Promising treatment. The larval stage of a disappointing one.

Calling treatments "promising" is a problem that seems to afflict all sides - including evidence-based medicine (EBM). As I'm lampooning the worst side of clinical experience with this cartoon though, it seems only fair for balance to have a shot at EBM at the same time. Here's where I've written about problems in EBM this year at MedPage Today and PLOS Blogs.

The most important article recently on this subject, though, is from Trisha Greenhalgh and colleagues: Six 'biases' against patients and carers in evidence-based medicine. 

(Cartoon spruced up and text added on 12 September 2015.)

Effectiveness delusions - don't become a statistic!

To inoculate yourself against "significant" effects that might not improve health, have a look at papers by Ioannidis and Gotzsche. Want to know more about the risks of relying only on biomarkers? Here's an explanation of their pitfalls at PubMed Health.