Promising = over-hyped + under-tested

We've apparently been using the word "promising" to mean "showing signs of future excellence" since about 1600. I first wrote about the tendency of "promising" treatments to metamorphose into "disappointing" treatments in a BMJ piece about evidence based mistakes. Early results, after all, can't promise anything at all.

For all sorts of reasons, research findings are themselves over-positive. That includes the most highly cited clinical studies in the "best" journals: in a study back in 2005, over 30% were judged to be contradicted or turned out to have over-estimated benefits; and in another in 2019, over 10% of randomized trials reversed previous findings.

My cartoon graph depicts a cumulative meta-analysis: each new study is being absorbed into a summation of the evidence so far. With 4 studies, it's shifted from the "this helps" side of the ledger over to the "this harms" side. See more about cumulative analyses in this classic article.

Speaking of classic articles, here's another: in the BMJ's 2015 tongue-in-cheek christmas issue, "promising" was one of the positive hype words Christiaan Vinkers and colleagues analyzed in PubMed's abstracts - from 1974 to 2014.

"Novel" was another favorite - it was one of the words with an increase of up to 15,000%: "At this rate, we predict that the word 'novel' will appear in every record by the year 2123"! A bold, innovative take! Novel was a major one in a study of rising hype in the abstracts of applications for NIH funding, too. In which the authors discuss the term "semantic bleaching": the overuse of hyperbole can bleach out a word's meaning (Millar 2022).

So is "promising" still increasing? Yep. It was in 2.4% of 2014's PubMed records with abstracts, 3.3% in 2016 – and 4.5% in 2021! (My calculations are included in this post.)

"Promising" is a media and marketing staple, too. In the last 24 hours, you can find "promising" results in Google News for kidney cancer "options", a skin antibiotic test, candidates for new antibiotics, leishmaniasis treatment, skin infections, a "novel" anti-tumor DNA vaccine, marijuana for Hepatitis C...

There used to be several initiatives worldwide aiming to keep the media to account on this, story by story. In 2022, though, I could only find a couple still going strong: Germany's Median-Doktor.de and Japan's Media Doctor.

The breathless hype marches relentlessly on. On the plus side, at least words like "promising" mean that sometimes at least, marketing or optimism bias comes clearly labeled!



This post was updated on 25 February 2017, adding the study and data on the use of "promising" in PubMed. It was updated again on 16 May 2020, adding the 2019 study on medical reversals. And again on 15 November 2022 updating the data on "promising" in PubMed, and adding the source for my calculations; adding the study of NIH abstracts; and deleting NHS' Behind the Headlines and the US Health News Review, which have been discontinued.