Drugs go head-to-head at the Pharma Olympics

At the Olympics, humans try to go "higher, faster, stronger" - and achieve their personal best. The bar is constantly raised. Drugs don't have to be better to cross the line, though: they can get by on what's called non-inferiority or equivalence trials. "No worse" (more or less) can be good enough.

Some drugs are now only loosely possibly non-inferior to other non-inferior drugs - several degrees removed from proven superior to doing nothing. Add the increasing reliance on shortcut measures of what works, and there's a real worry that for drugs, the performance bar is being lowered.

If you want to read about the differences between traditional randomized controlled trials that can show superiority and their non-inferiority and equivalence cousins, click on the PDF here at the CONSORT website.

Epidemiology jumps species

Technically, I guess Shelley would be a specialized type of epizoologist (or epizootiologist) - someone who studies patterns of diseases in animals.


Epidemiology comes from the word epidemic, which means visited upon humans. It was given its medical meaning by Hippocrates. But the study of epidemiology has become broader than only epidemics. And the profession is growing, too.


 "Clinical epidemiology" was proposed as a discipline by John Paul in 1938. He described it as concerned with a deeper understanding of the patient and the social determinants of health. The first textbooks on clinical epidemiology were published in the 1980s: you can read more about the history here.


Around the same time, the term "evidence-based medicine" came into vogue, a concept that had unfortunately shed the social determinants' focus along the way. Without a notion of "unbiased" embedded in it either, evidence-based medicine is in danger of becoming a label that can be applied to almost anything. But clinical epidemiology rocks on!

Blind luck: in praise of control groups

People often don't like the idea of "drawing the short straw" in a randomized trial. But being in the control group could turn out to be a very lucky break!

Ideally, trials are blinded so you don't know which group you're in. Knowing you're in the control group could affect your behavior and your opinions about whether or not you're benefiting (or being harmed). But even when it's not possible, it's not always fail-proof. (A placebo wasn't going to do THAT to Lisa's eyebrows!)

In theory, a trial is being done because it's genuinely not known whether the interventions being tested are better than alternatives (including doing nothing). And people who participate in clinical trials, on average, don't seem to be any worse off than people who don't - whether or not they were in a treatment or control group.

For studies that addressed this question, it was possible for researchers to get an average on experimental versus established treatments: only around half of new experimental treatments turned out to be better, and very few turned out to be a lot better. Those studies only covered about 1% of trials. Still, it's reassuring to know that people who participate in trials and end up in control groups aren't necessarily losing out.


If you'd like to read a quick introduction about control groups, go to the short sections 8.11 and 8.12 in Part 2 of the Cochrane Handbook. And here's research on blinded allocation to trials and on subjective assessment in trials.

Citation definitely needed

See also my guest blog at Scientific American:

"Are you a knowledge philanthropist? If not, why not?"

She's definitely picked up one of Wikipedia's most important messages [citation needed]: "Exercise caution before relying on unsourced claims." Even if you don't want to devote time to editing Wikipedia pages, why not learn to at least add the occasional (reliable) citation where it's needed? It's a far more important habit than clicking"like" buttons! Wikipedia's medical pages get well over 3 million visits a day.


Systematic reviews are designated as one of the reliable sources for medical claims in Wikipedia. Making better use of systematic reviews will be discussed by WikiProject Medicine at the international Wikipedia conference, Wikimania - kicking off tonight at the Library of Congress, Washington DC. Here's the abstract for my Wikimania talk.

Evidence-Based Chirping

With apologies to all birds: they don't really believe their pre-dawn chirping makes the sun come up!


This kind of thinking is one of the most common traps for humans about their health though - even for healthcare professionals and researchers. Just because two things happen at the same time, it doesn't mean one causes the other.

The secret life of trials....

When trials disagree...it can get ugly! But going into meta-analysis could help sort things out.

For a meta-analysis - a technique for combining the results of multiple trials - trials have to pretty much belong together. Differences might be responsible for contradictory results - including differences in the people in the trials, the way they were treated, or the way the trials were done. That's called heterogeneity. Too much of it, and the trials shouldn't be together. But heterogeneity isn't always a deal breaker.

Want to read more about heterogeneity in systematic reviews? Here's an article by Paul Glasziou and Sharon Sanders from Statistics in Medicine [PDF]. Or try the open learning materials from the Cochrane Collaboration.

Promising = over-hyped + under-tested

We've apparently been using the word "promising" to mean "showing signs of future excellence" since about 1600. I first wrote about the tendency of "promising" treatments to metamorphose into "disappointing" treatments in a BMJ piece about evidence based mistakes. Early results, after all, can't promise anything at all.

For all sorts of reasons, research findings are themselves over-positive. That includes the most highly cited clinical studies in the "best" journals: in a study back in 2005, over 30% were judged to be contradicted or turned out to have over-estimated benefits; and in another in 2019, over 10% of randomized trials reversed previous findings.

My cartoon graph depicts a cumulative meta-analysis: each new study is being absorbed into a summation of the evidence so far. With 4 studies, it's shifted from the "this helps" side of the ledger over to the "this harms" side. See more about cumulative analyses in this classic article.

Speaking of classic articles, here's another: in the BMJ's 2015 tongue-in-cheek christmas issue, "promising" was one of the positive hype words Christiaan Vinkers and colleagues analyzed in PubMed's abstracts - from 1974 to 2014.

"Novel" was another favorite - it was one of the words with an increase of up to 15,000%: "At this rate, we predict that the word 'novel' will appear in every record by the year 2123"! A bold, innovative take! Novel was a major one in a study of rising hype in the abstracts of applications for NIH funding, too. In which the authors discuss the term "semantic bleaching": the overuse of hyperbole can bleach out a word's meaning (Millar 2022).

So is "promising" still increasing? Yep. It was in 2.4% of 2014's PubMed records with abstracts, 3.3% in 2016 – and 4.5% in 2021! (My calculations are included in this post.)

"Promising" is a media and marketing staple, too. In the last 24 hours, you can find "promising" results in Google News for kidney cancer "options", a skin antibiotic test, candidates for new antibiotics, leishmaniasis treatment, skin infections, a "novel" anti-tumor DNA vaccine, marijuana for Hepatitis C...

There used to be several initiatives worldwide aiming to keep the media to account on this, story by story. In 2022, though, I could only find a couple still going strong: Germany's Median-Doktor.de and Japan's Media Doctor.

The breathless hype marches relentlessly on. On the plus side, at least words like "promising" mean that sometimes at least, marketing or optimism bias comes clearly labeled!



This post was updated on 25 February 2017, adding the study and data on the use of "promising" in PubMed. It was updated again on 16 May 2020, adding the 2019 study on medical reversals. And again on 15 November 2022 updating the data on "promising" in PubMed, and adding the source for my calculations; adding the study of NIH abstracts; and deleting NHS' Behind the Headlines and the US Health News Review, which have been discontinued.

Begging hopefully for less bias

From my guest blog post at Scientific American - Holy sacred cow!

Personal bias, wishful thinking plus biased research results is one recipe for a sacred cow. If more rigorous research results in a conflicting message, it could cause cognitive dissonance - and the less biased research often faces an uphill battle for acceptance.

And I also wrote about the importance of being just as rigorous about the claims we want to believe as those we're skeptical of here at the British Journal of Sports Medicine blog.

If you want to get better at critically assessing health claims, Smart Health Choices is a great place to start.

In 2018, the Centre for Evidence-Based Medicine at Oxford University began a Catalogue of Bias - with a blog and Twitter.


[Updated on 11 March 2018]

The trial acronymania menace

As if there's not enough for us to remember, we're supposed to remember endless acronyms for trials too now. There's even a wiki to help us keep them straight and a call for a register of trial acronyms to reduce multiple use of all the words ending in T!

Somewhere along the line this became marketing: not much equipoise in ACHIEVE, MIRACLE, PROMISE, or IMPROVE-IT, eh? A study has classified this as a form of coercion. Ivan Oransky called for a HALT (Help Acronyms Leave Trials). If you're irritated by the next outbreak of trial acronymania or acronymesis you come across, you're not alone!


Another trial acronym here at Statistically Funny.....Meet the AGHAST Investigators!



(Updated with IMPROVE-IT on 16 July 2019.)

Why oh why not randomize?