tag:blogger.com,1999:blog-63530975538199346242024-03-14T14:49:04.840-04:00Statistically FunnyCommenting on the science of unbiased health research with cartoonsHilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.comBlogger66125tag:blogger.com,1999:blog-6353097553819934624.post-2186057695868092742023-06-24T01:36:00.002-04:002023-06-24T02:23:39.315-04:00A study in misplaced scientific flair<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizeg6lSwN4UpHoZWestHJTMS0Gw1dkZ9e675_HMopRM8x0jb_iFEjhSYN0ik8W2u0OYqPtCYaBQU1rGVUjZaRBwRE-vl4UnmhvD5jt447W36CrDn7fjKDP_ptTRuuW2v6SvaJA8Qo1CPEeRfwWwR_6TqqAXFfqbrkethUJYqlHiDK7xXoWyO8Kta6lJhVT/s2485/Do%20an%20abstract.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="A senior and junior scientist are looking at a colorful painting on the lab wall. "What is it?," asks the older one. "You said to do an abstract," answers the young one." border="0" data-original-height="1667" data-original-width="2485" height="339" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizeg6lSwN4UpHoZWestHJTMS0Gw1dkZ9e675_HMopRM8x0jb_iFEjhSYN0ik8W2u0OYqPtCYaBQU1rGVUjZaRBwRE-vl4UnmhvD5jt447W36CrDn7fjKDP_ptTRuuW2v6SvaJA8Qo1CPEeRfwWwR_6TqqAXFfqbrkethUJYqlHiDK7xXoWyO8Kta6lJhVT/w504-h339/Do%20an%20abstract.jpg" width="504" /></a></div><p><br /></p>It's the most-read part of scientific articles. Abstracts are supposed to give you a quick impression of research results. But while they may be small, they're crammed with traps. Hype in scientific articles <a href="https://absolutelymaybe.plos.org/2022/08/30/bad-and-good-ish-news-on-the-abstract-spin-cycle/">seems to be escalating</a>, and abstracts – those little study blurbs – tend to be the hypey-est part.<p></p><p>Of course, sometimes scientists have already gotten creative with a biased title for the article – bonus "points" for getting in first even earlier, with the study name or its acronym. (More on the acronymania menace <a href="https://statistically-funny.blogspot.com/2012/06/trial-acronymania-menace.html">here at <i>Statistically Funny</i></a>.)</p><p>What does this mean for us as readers? It means we can't be sure about the real takeaway messages from a study based on the abstract alone. Which, frankly, sucks. </p><p>There's some good news at least: There's <a href="https://absolutelymaybe.plos.org/2022/08/30/bad-and-good-ish-news-on-the-abstract-spin-cycle/">some serious research</a> into the problem. And there are some telltale signs. I reckon you have to keep a sharp eye out for adjectives and adverbs – that's where things often take a spicy turn.</p><p>In May 2023, Olivier Corneille and colleagues <a href="https://elifesciences.org/articles/88654">published</a> a list of 22 persuasive communication devices to watch out for in academic papers. Gulp! There's a summary of the list in <a href="https://elifesciences.org/articles/88654#table1">table 1</a>.</p><p>On the plus side, today's hype words mightn't work for long, thanks to "semantic bleaching". That's when overuse of hyperbole <a href="https://absolutelymaybe.plos.org/2022/08/30/bad-and-good-ish-news-on-the-abstract-spin-cycle/">"'bleaches' out the stronger meaning of the word."</a> Though I guess there will always be new buzz words to take their place. Sigh!</p><p><br /></p><p><br /></p><p></p><ul style="text-align: left;"><li><a href="https://absolutelymaybe.plos.org/2018/06/06/building-a-great-scientific-abstract-a-quick-checklist/">My quick checklist</a> for building a great scientific abstract.</li><li>Other <i>Absolutely Maybe </i><a href="https://absolutelymaybe.plos.org/?post_tag=abstracts">posts tagged "abstracts"</a>.</li></ul> <p></p><p><i>Disclosure: I'm a co-author of the 2013 <a href="https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001419">PRISMA reporting guidelines</a> for abstracts of systematic reviews. I have unfortunately contributed to the huge pile of conference abstracts that have never been followed by published papers, including a couple that evaluated the quality of abstracts (<a href="https://abstracts.cochrane.org/1998-baltimore/quality-cochrane-systematic-review-abstracts-readability-and-comparison-handbook">1998a</a>, <a href="https://abstracts.cochrane.org/1998-baltimore/quality-assurance-cochrane-systematic-review-abstracts-comparison-abstracts-published">1998b</a>).</i></p>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-18038938129210660832022-12-14T01:22:00.002-05:002022-12-14T15:31:02.456-05:00So, so many questions!<p style="text-align: justify;"><span style="font-family: inherit;"> </span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjniLyl4tZZJyUvouI7YEQgglDefSNAAihCxvCO8tTFAA_rNj0CMz3qp02Bz5YhKqBTjbowg8AXBNSMnelmlEkrErwSGySXEUN5QH0Qnj2l5LHact1YvAk_PCrjMvI6raks7X8HS5JMrAK7xkXbBAymzzmIGCZBi5EDUj_MH0Q6pfob0Ay_Jj4wUGQuuQ/s600/Science%20formula%202.jpg" style="margin-left: 1em; margin-right: 1em;"><span style="color: black; font-family: inherit;"><img alt="Cartoon blackboard. Science is a method for increasing the number of questions. Expressed by the formula: ? + science = (? + ?) squared" border="0" data-original-height="263" data-original-width="600" height="175" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjniLyl4tZZJyUvouI7YEQgglDefSNAAihCxvCO8tTFAA_rNj0CMz3qp02Bz5YhKqBTjbowg8AXBNSMnelmlEkrErwSGySXEUN5QH0Qnj2l5LHact1YvAk_PCrjMvI6raks7X8HS5JMrAK7xkXbBAymzzmIGCZBi5EDUj_MH0Q6pfob0Ay_Jj4wUGQuuQ/w400-h175/Science%20formula%202.jpg" width="400" /></span></a></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">I first encountered science seriously as a health consumer advocate, a very long time ago. And I thought of medical and health research as a search for answers. Scientists were problem-solvers, using rigorous testing to sort out the wheat of reliable answers from the chaff of the false leads.</span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">But over time, as I watched the research pile up exponentially, the number of questions was zooming up even faster.</span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">We find out treatment <i>A</i> works. Who – and what – else could it work for? Can you have half the dose and still get just as much benefit? Does double the dose do more good than harm? Will it work even better if you combine it with treatment <i>B</i>? Combine it with treatment <i>C</i>? Combine it with <i>B</i> <b><i>and</i></b> <i>C</i>? Is it better than old treatment <i>Q</i>? Will work it in gel form? In spray? . . .<span class="Apple-converted-space"> </span></span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">Turns out scientific studies are, in fact, a great way to generate more questions. Answers, on the other hand, are often elusive.</span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">So whose questions does science work with? For most of science’s history, the work has been dominated by one gender and one race, from just a few countries. The barriers to scholars from the Global South and access to, and recognition of, <a href="https://direct.mit.edu/qss/article/doi/10.1162/qss_a_00228/114119/Recalibrating-the-Scope-of-Scholarly-Publishing-A">their work internationally</a> remain shockingly high. On top of that, scientists typically worked at arm’s length from the people affected by the problems they were trying to solve. The result was often a very narrow set of questions. </span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">Consider the <a href="https://bmjopen.bmj.com/content/3/5/e002241">experience of clinical researchers in rheumatology</a>. It wasn’t until after the field embraced consumer participation that fatigue, sleep, and disease flares were seen as important outcomes to measure. And the process changed their research culture, too.</span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">Diversity in science is critical, too, to bring a variety of perspectives to the question-asking table. That includes diversity of disciplines in scientific teams. Working across disciplines doesn’t just bring different points of views into the process. It can be essential for scientific quality. Poor scientific methods that have been rejected in some disciplines, persist in other parts of academia. Interdisciplinary science might be able to <a href="https://osf.io/preprints/metaarxiv/cm5v3/">spread superior scientific methods</a> into fields with weaker science.<span class="Apple-converted-space"> </span></span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span style="font-family: inherit;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;">If questions are so important to how our knowledge grows – and they are – then the issue of who gets to ask those questions is a fundamental concern. It can have a profound impact on what questions get addressed at all, and what is seen.</span></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;"><br /></span></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;"><br /></span></span></p><p class="p1" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span class="s1" style="font-kerning: none;"><span style="font-family: inherit;"><i>(This post is based on a <a href="https://absolutelymaybe.plos.org/2019/05/29/science-a-method-for-increasing-the-number-of-questions/">2019 post</a> at Absolutely Maybe, drawing on the point on interdisciplinary science elaborated on in a <a href="https://hildabastian.substack.com/p/a-word-stretched-far-beyond-its-breaking">2022 post</a> at Living With Evidence.)</i></span></span></p><p class="p2" style="-webkit-text-stroke-color: rgb(15, 29, 54); color: #0f1d36; font-family: Helvetica; font-size: 16px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 19px; text-align: start;"><span class="s1" style="font-kerning: none;"></span></p></div>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-50712579103806643212022-11-21T20:14:00.001-05:002022-11-21T21:40:52.921-05:00Some studies are MONSTERS!<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU9xVRBfh90M8sJwXz7KT9q705uaAIEti0UUPUVNTciA2DRVH10VRICpmslELZt4a0-k6c3qenmRQjD-baqfNJsuqxgN0TtjqS5pWaRn8Xh41WX6u-A5QWeuwmybqicfTh1taU8NUEK7zVxqNwVmpMxKkhEDvnGwdPxGCSKH9xrVTcfVkw44UyFGeJ8A/s598/That-jerk.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="Cartoon of 2 small studies on one side of a meta-analysis, with a very big 3rd study on the other side pulling the studies' combined result over to his side. One of the little studies is thinking "That jerk is always throwing his weight around!"" border="0" data-original-height="598" data-original-width="587" height="415" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU9xVRBfh90M8sJwXz7KT9q705uaAIEti0UUPUVNTciA2DRVH10VRICpmslELZt4a0-k6c3qenmRQjD-baqfNJsuqxgN0TtjqS5pWaRn8Xh41WX6u-A5QWeuwmybqicfTh1taU8NUEK7zVxqNwVmpMxKkhEDvnGwdPxGCSKH9xrVTcfVkw44UyFGeJ8A/w409-h415/That-jerk.jpg" width="409" /></a></div><br /><p></p><p><span style="color: #112846;"><span style="font-family: inherit;">On the plus side, this jerk explains a lot about the data in a meta-analysis!</span></span></p><p>This cartoon is a forest plot, a style of data visualization for meta-analysis results. Some people call them "blobbograms". Each of these horizontal lines with a square in the middle represents the results of a different study. The length of that horizontal line represents the length of the confidence interval (CI). That gives you an estimate of how much uncertainty there is around that result - the shorter it is, the more confident we can be about the result. (<i>Statistically Funny</i> explainer <a href="http://statistically-funny.blogspot.com/2013/02/you-will-meet-too-much-false-precision.html">here</a>.)</p><p>The square is called the point estimate - the study's "result" if you like. Often, it's sized according to how much weight the study has in the meta-analysis. The bigger it is, the more confident we can be about the result.</p><p>The size of the point estimate is echoing the length of the confidence interval. They are two perspectives on the same information. Small square and long line provides less confidence than a big square with a short line.</p><p>
</p><p><br /></p><br /><div class="separator" style="clear: both; text-align: center;"><p style="-webkit-font-smoothing: antialiased; box-sizing: border-box; color: #112846; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 16px; line-height: 1.5; margin-left: 1em; margin-right: 1em; margin-top: 0px; text-rendering: optimizelegibility;"><img alt="Cartoon showing a big smirking cartoon dragging the summary estimate diamond over to his side of the meta-analysis" class="aligncenter size-full wp-image-19779" height="139" src="https://absolutelymaybe.plos.org/wp-content/uploads/sites/8/2020/05/Summary-estimate.jpg" width="253" /></p></div><div><p style="-webkit-font-smoothing: antialiased; box-sizing: border-box; color: #112846; line-height: 1.5; margin: 0px 0px 1.875rem; text-rendering: optimizelegibility;"><br /></p><p>The diamond here is called the summary estimate. It represents the summary of the results from the 3 studies combined. It doesn't just add up the 3 results then divide them by 3. It's a weighted average. Bigger studies with more events count for more. (More on that later.)</p><p>The left and right tips of the diamond are the two ends of the confidence interval. With each study that gets added to the plot, those tips will get closer together, and it will move left or right if a study's result tips the scales in one direction.</p><p>The vertical line in the center is the "line of no effect". If a result touches or crosses it, then the result is not statistically significant. (That's a tricky concept: my explainer <a href="https://blogs.plos.org/absolutely-maybe/2016/04/25/5-tips-for-avoiding-p-value-potholes/">here</a>.)</p><p>In biomedicine, forest plots are the norm. But in other fields, like psychology, the results of meta-analyses are often presented as tables of data. That means that each data point - the start and end of each confidence interval, and so on - are numbers in a column instead of plotted on a graph. (<a href="https://www.researchgate.net/profile/Wendy_Phillips3/publication/282609096_Thinking_Styles_and_Decision_Making_A_Meta-Analysis/links/580ffafd08aef2ef97afecb1/Thinking-Styles-and-Decision-Making-A-Meta-Analysis.pdf">Here's a study</a> that does that.)</p><p>
</p><p>So what about that jerk? He carries so much weight not just because the study has a lot of participants in it. What's called a study's precision depends on the number of "events" in the study, too. </p><p><span style="color: #112846; font-size: 16px;"><span style="font-family: inherit;">Say the event you’re interested in is heart attacks – and you are investigating a method for reducing them. But for whatever reason, not a single person in the experimental or control group has a heart attack even though the study was big enough for you to have expected several. That study would have less ability to detect any difference your method could have made, so the study would have less weight.</span></span></p><p>It's very common for a study, or a couple of them, to carry most of the weight in a meta-analysis. A study by Paul Glasziou and colleagues found that the trial with the most precision carried an average of 51% of the whole result. When that's the case, you really want to understand that study.</p><p>Some studies are such whoppers that they overpower all other studies – no matter how many of them there are. They may never be challenged, just because of their sheer size: No one might ever do a study that large on the same question again.</p><p>The size of the point estimate and length of the line around it are clues to the weight of the study. The meta-analysis might also include the percentages of weight for each study.</p><p>Like to know more? This is a shorter version of one of the tips in my post at <i>Absolutely Maybe</i>, <a href="https://absolutelymaybe.plos.org/2017/07/03/5-tips-for-understanding-data-in-meta-analyses/">5 Tips for Understanding Data in Meta-Analyses</a>. Check it out for a more in-depth example of looking at the weight of a study and 4 more key tips!</p><p><i>Hilda</i></p></div>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-39000987801394325392022-10-31T16:42:00.001-04:002022-10-31T16:42:10.242-04:00Researching our way to better research?<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ9WDDgbgaCegPu5G8yAggiLYV4oaFerHGScWCS05qJ_1G17VrNPjqaCQQKvgK1RkRtMdLGIeTy4QX3pYzT0FKZ0j_xZYddr0Ym4oXgjcD3-o94p48_aAXK-4B_3yozPKjzwLSfaYppjXSmI1cb2IOMfvqkqWK9sWTN2Q5g4ZV53LUXTVz2nYB0GAY6Q/s955/Research-on-research.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="Cartoon: I do research on research. Person 2: Terrific! I research the research of research" border="0" data-original-height="837" data-original-width="955" height="350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ9WDDgbgaCegPu5G8yAggiLYV4oaFerHGScWCS05qJ_1G17VrNPjqaCQQKvgK1RkRtMdLGIeTy4QX3pYzT0FKZ0j_xZYddr0Ym4oXgjcD3-o94p48_aAXK-4B_3yozPKjzwLSfaYppjXSmI1cb2IOMfvqkqWK9sWTN2Q5g4ZV53LUXTVz2nYB0GAY6Q/w400-h350/Research-on-research.jpg" width="400" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">Here we see an expert in evidence synthesis meet a metascientist!</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Evidence synthesis is an umbrella term for the work of finding and making sense of a body of research – methods like <a href="https://statistically-funny.blogspot.com/search/label/Reviews%20%26%20meta-analysis">systematic reviews and meta-analysis</a>. And metascience is studying the methods of science itself. It includes studying the way science is published – see for example <a href="https://absolutelymaybe.plos.org/?category=peer-review-research-roundup">my posts on peer review research</a>. And yes, there's metascience on evidence synthesis, too – and syntheses of metascience!</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">The terms <a href="https://en.wikipedia.org/wiki/Metascience">metascience</a> and <a href="https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002264">metaresearch</a> haven't been tossed around for all that long, compared to other types of science. Back when I took my first steps down this road in the early 1990s, in my neck of the science woods we called people who did this methodologists. A guiding light for us was the statistician and all-round fantastic human Doug Altman (<a href="https://www.bmj.com/content/361/bmj.k2588">1948-2018</a>). He wrote a rousing editorial <a href="https://www.bmj.com/content/308/6924/283">in 1994</a> called "The scandal of poor medical research," declaring "We need less research, better research, and research done for the right reasons." Still true, of course.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Altman and colleague, Iveta Simera, chart the early history of metascience over at the <a href="https://www.jameslindlibrary.org/articles/a-history-of-the-evolution-of-guidelines-for-reporting-medical-research-the-long-road-to-the-equator-network/">James Lind Library</a>. Box 1 in that piece has a collection of scathing quotes about poor research methodology, starting in 1917 with this one on clinical evidence: "A little thought suffices to show that the greater part cannot be taken as serious evidence at all."</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">The first piece of research on research that they identified was published – with only the briefest of detail, unfortunately – by Halbert Dunn in 1929. He analyzed 200 quantitative papers, and concluded, "About half of the papers should never have been published as they stood." (It's on the second page <a href="https://journals.physiology.org/doi/pdf/10.1152/physrev.1929.9.2.275?casa_token=8JGA5jyLoccAAAAA:KHrOVLl9o2cviJo95UFTTG4sH4Yi7MPetp0XkuFLYwjuz_51JsxZXwFt3I-uGe6ZRiNVSus8jQk">here</a>.)</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">The first detailed report came <a href="https://jamanetwork.com/journals/jama/article-abstract/658800?casa_token=F7Ap6Q591soAAAAA:RwGc8m2j5woUYh28xlMHCnMlXIp2XLY3p4P_iNx-KqhtZQZ9NN9dPwR81GnEktr7K9cbUo6M">in 1966</a>, by a statistician and medical student. They reckoned over 70% of the papers they examined should either have been rejected or revised before being published. A few years after that, the methods for evidence synthesis took an <a href="https://www.jameslindlibrary.org/light-rj-smith-pv-1971/">important step forward</a> when Richard Light and Paul Smith published their "procedures for resolving contradictions among different research studies" (<a href="https://meridian.allenpress.com/her/article-abstract/41/4/429/30841/Accumulating-Evidence-Procedures-for-Resolving">Light and Smith, 1971</a>.)</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Evidence synthesis and metascience have proliferated wildly since the 1990s. And there's lots of the better research that Altman hoped for, too. Unfortunately, though, it's still in the minority – even in evidence synthesis. Sigh! Will more research on research help? Someone should do research on that!</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><i>Hilda Bastian</i></div><div class="separator" style="clear: both; text-align: left;"><i><br /></i></div><div class="separator" style="clear: both; text-align: left;"><br /></div><p></p>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-80028283045804922772022-10-11T02:47:00.005-04:002022-10-11T02:50:07.922-04:00Trial participants and the luck of the draw<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCYYIRG-_0SMN73lkHuDXhqqbsW_GCj1YVGOwtv-7qmgKTfSsZRSNoRdYV3WWE70My7LDE1GYg7MKi5gr5VrnMfvCrCcq05wMEeRpJFga0vnIe0ohQcDuHpTSwyIxA0yARLv41_uYZZkhlEN5jdRmjFvbYn7CCWbDkuRtqG_oThx7BKiL4GkqNjCG7ug/s4599/Study-result-sharing.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="Cartoon: Any questions about the study results? Surprised person thinking "I was in a study?!"" border="0" data-original-height="3840" data-original-width="4599" height="334" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCYYIRG-_0SMN73lkHuDXhqqbsW_GCj1YVGOwtv-7qmgKTfSsZRSNoRdYV3WWE70My7LDE1GYg7MKi5gr5VrnMfvCrCcq05wMEeRpJFga0vnIe0ohQcDuHpTSwyIxA0yARLv41_uYZZkhlEN5jdRmjFvbYn7CCWbDkuRtqG_oThx7BKiL4GkqNjCG7ug/w400-h334/Study-result-sharing.jpg" width="400" /></a></div><br /><p></p><p>The guy in this cartoon really drew a short straw: <a href="https://sigmapubs.onlinelibrary.wiley.com/doi/10.1111/jnu.12097">most clinical trial participants</a>, at least, know they were in a study. On the other hand, he was lucky that he was getting to hear from the researchers about the study's results! That <a href="https://absolutelymaybe.plos.org/2016/12/22/silence-everyday-betrayals-of-research-participants/">used to be</a> quite unlikely.</p><p>It might be getting better: <a href="https://bmjopen.bmj.com/content/9/10/e032701.abstract">a survey</a> of trial authors from 2014-2015 found that half said they'd communicated results to participants. That survey had a low response rate – about 16% – so it might not be the best guide. There are quite a few studies these days on how to communicate results to participants, though, and that could be a good sign. (A systematic review of those studies is <a href="https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-019-1065-x">on the way</a>, and I'll be keeping an eye out for it.)</p><p>Was our guy lucky to be in a clinical trial in the first place, or was he taking on a serious risk of harm?</p><p>An older review of trials (<a href="https://www.cmaj.ca/content/186/16/E596.short">up to 2010</a>) across a range of diseases and interventions found no major difference: trial participants weren't apparently more likely to benefit or be harmed. Another in women's health trials (<a href="https://obgyn.onlinelibrary.wiley.com/doi/full/10.1111/1471-0528.14528">up to 2015</a>) concluded women who participated in clinical trials did better than those who didn't. And a recent one in pregnant women (<a href="https://www.sciencedirect.com/science/article/abs/pii/S2589933322001276">up to May 2022</a>) concluded there was no major difference. All of this, though, relies on data from a tiny proportion of all the trials that people participate in – and we don't even know the results of many of them.</p><p>I think a really thorough answer to this question would have to differentiate the types of trials. For perspective, consider clinical trials of drugs. <a href="https://academic.oup.com/biostatistics/article/20/2/273/4817524?login=false">Across the board</a>, roughly 60% of drugs that get to phase 1 (very small early safety trials) or phase 2 (mid-stage small trials) don't make it the next phase. Most of the drugs that get to phase 3 (big efficacy trials) end up being approved: over 90% in 2015. The rate is higher than average for vaccines, and much lower for drugs for some diseases than others.</p><p>Not progressing to the next stage doesn't tell us if people in the trials benefited or were harmed on balance, but it shows why the answer to the question of impact on individual participants could be different for different types of trials.</p><p>So was the guy in the cartoon above lucky to be in a clinical trial? The answer is a very unsatisfactory, it depends on his specific trial! However, overall, there's no strong evidence of benefit or harm. </p><p>On the other hand, not doing trials at all would be a very risky proposition for the whole community. No matter which way you look at it, the rest of us have a lot of reasons to be very grateful to the people who participate in clinical trials: thank you all!</p><p><br /></p><p><i>If you're interested in reading more about the history of people claiming either that participating in clinical trials is inherently risky or inherently beneficial, I dug into this in a post at Absolutely Maybe <a href="https://absolutelymaybe.plos.org/2020/10/31/clinical-trials-are-vital-for-the-rest-of-us-are-they-a-good-deal-for-the-participants/">in 2020</a>. </i></p><p><br /></p>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com1tag:blogger.com,1999:blog-6353097553819934624.post-17492130179999676022022-10-03T00:17:00.005-04:002022-10-03T00:24:22.835-04:00How are you?<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge6bcCaHk6yZ0o6naCOGBoKbpk4ZVbZkPnG9PWWGNrqtTX1VUgYK2lJuc6_-TR0lz4UHNkR407SjVZTycroQjPQfy0xJhgqzdnPz4Akxtyl_O66ZlZ0nq4md3KsvQ4Q1fFsioYB2Bn8ZX_t-lEWzFwYpFyjfOZl4HEqHItgRV4VKW84flF4m8kWdgGTA/s2821/Half-a-standard-deviation-updated.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="Cartoon of a person answering the question how are you with "about half a standard deviation below the mean"" border="0" data-original-height="1635" data-original-width="2821" height="279" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge6bcCaHk6yZ0o6naCOGBoKbpk4ZVbZkPnG9PWWGNrqtTX1VUgYK2lJuc6_-TR0lz4UHNkR407SjVZTycroQjPQfy0xJhgqzdnPz4Akxtyl_O66ZlZ0nq4md3KsvQ4Q1fFsioYB2Bn8ZX_t-lEWzFwYpFyjfOZl4HEqHItgRV4VKW84flF4m8kWdgGTA/w483-h279/Half-a-standard-deviation-updated.jpg" width="483" /></a></div><br /><span><br /></span><div><span style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;">A simple question, theoretically, has a simple answer. That's not necessarily the case in a clinical trial, though. <span style="color: #112846; font-size: 16px;">To measure in a way that can detect differences between groups, researchers often have to use methods that bear no relationship to how we think of a problem, or usually describe it.</span></span></div><div><span style="font-family: inherit;"><span style="color: #112846; font-size: 16px;"><br /></span></span></div><div><span style="font-family: inherit;"><span style="color: #112846; font-size: 16px;">Pain is a classic example. We use a lot of vivid words to try to explain our pain. But in a typical health study, that will be standardized. If that were done with what's called a "dichotomous" outcome – a straight up "yes or no" type question – it can be easy to understand the result.</span></span></div><div><br /></div><div>But outcomes like pain can be measured on a scale, which is a "continuous" outcome: how bad is that pain, from nothing to the worst you can imagine? By the time average results between groups of people's scores get compared, it can be hard to translate back into something that makes sense. That's what the woman in the cartoon here is doing: comparing herself to people on a scale.</div><div><br /></div><div><span style="font-family: inherit;"><span style="color: #112846; font-size: 16px;">It pays to never put too much weight on the name of an outcome – check what it really means in the context of that study. There could be fine print that would make a difference to you – for example, “mortality” is measured, but only in the short-term, and you might have to dig hard to figure that our. </span><span style="color: #112846; font-size: 16px;">Or the name the outcome is given might not be what it sounds like at all. People use the same names for outcomes they measure very differently.</span></span></div><div><span style="color: #112846; font-size: 16px;"><br /></span></div><div><span style="color: #112846; font-size: 16px;">Even something that sounds cut and dried can be...complicated. “Perinatal mortality” – death around childbirth – starts and ends at different times before and after birth, from country to country. “Stroke” might mean any kind, or some kinds. And then there's the complexity of composite outcomes – where multiple outcomes are combined and treated as if they're a single one. More on that <a href="https://statistically-funny.blogspot.com/2015/02/lets-play-outcome-mash-up-clinical.html">here at Statistically Funny</a>.</span></div><div><span style="color: #112846; font-size: 16px;"><br /></span></div><div><span style="color: #112846; font-size: 16px;">Some researchers put in the hard work of interpreting study measures to make sense in human terms. It would help the rest of us if more of them did that!</span></div><div><br /></div><div><span style="color: #112846; font-family: inherit;"><i><a href="https://statistically-funny.blogspot.com/search/label/Outcomes">More posts on outcomes at Statistically Funny</a></i></span></div><div><span style="color: #112846; font-family: inherit;"><br /></span></div><div><span style="color: #112846; font-family: inherit;"><i><a href="https://statistically-funny.blogspot.com/2013/04/dont-worry-its-just-standard-deviation.html">And what's a standard deviation from the mean?</a></i></span></div><div><span style="color: #112846; font-size: 16px;"><br /></span></div><div><span style="color: #112846; font-size: 16px;"><i><br /></i></span></div><div><span style="color: #112846; font-size: 16px;"><i>This post is based on a section of <a href="https://absolutelymaybe.plos.org/2017/12/21/6-tips-for-deciphering-outcomes-in-health-studies/">a post</a> on deciphering outcomes in clinical trials at my Absolutely Maybe blog.</i></span></div><div><span style="color: #112846; font-size: 16px;"><i><br /></i></span></div><div><span style="color: #112846; font-size: 16px;"><i>Hilda Bastian</i></span></div><div><span style="color: #112846; font-size: 16px;"><i><br /></i></span></div>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-17635846777036746202021-03-31T19:53:00.002-04:002022-10-02T03:47:54.761-04:00In clinical trials, you can have it both ways<p><br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwQEkFhTlrax3kwhKonONyW5PQZSOXOOcG60Nfx7bNEUemhU9YXNsTr3JkA7QMJezHnNuY_5lvCesJijvLUlWoEZej3rLP234a0z2tWJ6qQy_ndl832EtF7KfJM5Gh7-9hsRp-1csV3BRc/s2048/Cross-over_trials.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="Cartoon of people talking about being in the vaccine and placebo group" border="0" data-original-height="2048" data-original-width="1415" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwQEkFhTlrax3kwhKonONyW5PQZSOXOOcG60Nfx7bNEUemhU9YXNsTr3JkA7QMJezHnNuY_5lvCesJijvLUlWoEZej3rLP234a0z2tWJ6qQy_ndl832EtF7KfJM5Gh7-9hsRp-1csV3BRc/w442-h640/Cross-over_trials.jpg" width="442" /></a></div><br /> <p></p><p>"Were you in the vax group or the placebo?" <span style="font-family: Cambria;">It sounds like a simple question, that should have a simple answer, right? And usually it does. Unless it doesn’t. Welcome to the world of the cross-over trial!</span></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">The garden variety randomized trial is a parallel or concurrent trial: people get randomized to one of 2 or more groups, and they continue on their parallel tracks, at the same time. At the end of it, if all goes well, you have solid answers to the main question or questions you set out to resolve.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">In a cross-over trial, on the other hand, people start off in one group, then along the way, each group of people swaps over with those in another group. Everyone gets the same options, they are just randomized to going through them in a different order – intervention A then B, or intervention B then A. That’s how the guy in the cartoon can be in both the vaccine and the placebo group.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3d_5zR9ZuCAi5QPNGpM-seVDveOOOfEKAXCXT37tlwutpIcMLcN9syQershEHDVKNYlIl6-NCCN2vVG6dLqbUb6BNXHxiFu8gjuDuR2BmJzfgfOqYuvT_tH9oz_7l8o48jp0MWfEbOUsF/s1414/Crossover-trial.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Graph showing crossing over between A and B" border="0" data-original-height="489" data-original-width="1414" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3d_5zR9ZuCAi5QPNGpM-seVDveOOOfEKAXCXT37tlwutpIcMLcN9syQershEHDVKNYlIl6-NCCN2vVG6dLqbUb6BNXHxiFu8gjuDuR2BmJzfgfOqYuvT_tH9oz_7l8o48jp0MWfEbOUsF/w640-h222/Crossover-trial.jpg" width="640" /></a></div><br /><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Let's start with the <a href="https://training.cochrane.org/handbook/current/chapter-23#section-23-2">advantage</a> of doing trials like this. A crossover trial means each person is their own control. With that one move, you have removed a common reason for differences in outcomes – individuals' differences. And that means you need fewer people to get an answer.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Every health intervention question can't be answered this way of course – think surgery versus antibiotics for appendicitis, for example, or a drug that isn't going to leave your body to revert back to your usual state during the break between interventions ("wash-out period").</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">But what about our guy in a vaccine trial, though? They don't fit this picture, do they? Vaccines may wash out, but the benefits to your immune system of recognizing its enemy sure isn't supposed to!</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Crossover trials for vaccines are in the spotlight because they're being used for Covid-19 vaccine trials. I discuss this in depth <a href="https://absolutelymaybe.plos.org/2021/03/31/the-pioneering-cross-over-trials-for-covid-vaccines-and-what-well-find-out/">over at Absolutely Maybe</a> – and for more technical discussion on this, see <a href="https://www.medrxiv.org/content/10.1101/2020.12.14.20248137v1">this preprint</a> on the thinking behind the proposal, and <a href="https://www.fda.gov/media/144582/download">Steve Goodman's slides</a> for the US Food and Drug Administration's deliberations.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">The crossover extensions of the Covid vaccine trials can't do everything a randomized controlled trial can do, but they can provide valuable data on some issues, especially if the people stay blinded. High amongst those is how long immunity lasts. That's because you now have one group that was vaccinated early, and one group who had deferred vaccination. After the crossover, if the infection rate between the groups stays the same, you know the early-vaccinated group's immunity isn't waning.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Back to the average crossover trial, though, which will be of treatments. What should look out for with those?</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">One problem is if the groups before the crossover are treated as though they are parallel trials. That's risky. Randomizing enough people to a parallel trial means you don't have to worry about differences between the individuals skewing the results – you don't have that when you're randomizing the order of interventions, not the people.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">You also have to keep in mind what possible influence could the previous intervention have had. If the trial goes on for a while, then you have to consider whether the different time periods are now a factor – and more people might have dropped out before they had the second intervention, too.</p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">And 2 final bonus points: "N of 1" trials are cross-overs. That's when you are trying out treatments in a formally structured way, though like all cross-over trials, it only works in some situations. (A quick look at those <a href="https://statistically-funny.blogspot.com/2012/11/the-one-about-ship-wrecked.html">here at <i>Statistically Funny</i></a>.) And there's another kind of trial where people are controls for themselves: (<a href="https://statistically-funny.blogspot.com/2013/09/more-than-one-kind-of-self-control.html">Here's my quick look at those</a>.)</p><p class="p1" style="font-family: Cambria; font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-family: Cambria; font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><i>Hilda Bastian</i></div><div class="separator" style="clear: both; text-align: left;"><i>March 2021</i></div><div class="separator" style="clear: both; text-align: left;"><i><br /></i></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><i>To learn more about crossover trials, check out Stephen Senn's book, Cross-Over Trials in Clinical Research. <a href="https://www.worldcat.org/title/cross-over-trials-in-clinical-research/oclc/884637808">This link</a> will help you find it in a library near you.</i></div><div class="separator" style="clear: both; text-align: left;"><i><br /></i></div><div class="separator" style="clear: both; text-align: left;"><br /></div><p></p>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-15345788536823597552018-08-12T19:30:00.001-04:002022-10-02T03:48:24.275-04:00Clinical Trials - More Blinding, Less Worry!<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1yZRNj8uk2Lsv1_HGL2c9b32iUp1B_C97C1bt1nuFzh2np37G800i0pNaRUPBT4JLnOBWFCTVVVu42p5RTvajGkMC15Ozu3BdjaYkWObk-tlA4vTz94mXtZy5jTXa1mp4BpwWhyphenhyphenZEj6qC/s1600/Yoga+class+worries.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1253" data-original-width="920" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1yZRNj8uk2Lsv1_HGL2c9b32iUp1B_C97C1bt1nuFzh2np37G800i0pNaRUPBT4JLnOBWFCTVVVu42p5RTvajGkMC15Ozu3BdjaYkWObk-tlA4vTz94mXtZy5jTXa1mp4BpwWhyphenhyphenZEj6qC/s640/Yoga+class+worries.jpg" width="469" /></a></div>
<br />
<br />
<br />
She's right to be worried! There are <i>so</i> many possible cracks that bias can seep through, nudging clinical trial results off course. Some of the biggest come from people knowing which comparison group a participant will be, or has been, in. Allocation concealment and blinding are strategies to reduce this risk.<br />
<br />
Before we get to that, let's look at the source of the problems we're aiming at here: people! They bring subjectivity to the mix, even if they are committed to the trial - and not everyone who plays a role will be supportive, anyway. On top of that, randomizing people - leaving their fate to pure chance - can be the rational and absolutely vital thing to do. But it's also <a href="https://www.thelancet.com/journals/lancet/article/PIIS0140673696012019/fulltext" target="_blank">"anathema to the human spirit"</a>, so it can be awfully hard to play totally by the rules.<br />
<br />
And we're counting on a lot of people here, aren't we? There are the ones who enter an individual into one of the comparison groups in the trial. There are those individual participants themselves, and the ones dealing with them during the trial - healthcare practitioners who treat them, for example. And then there are the people measuring outcomes - like looking at an x-ray and deciding if it's showing improvement or not.<br />
<br />
What could possibly go wrong?!<br />
<br />
Plenty, it turns out. Trials that don't have good guard rails for concealing group allocation and then blinding it are likely to exaggerate the benefits of health treatments (meta-research on this <a href="https://www.bmj.com/content/336/7644/601" target="_blank">here</a> and <a href="https://academic.oup.com/aje/article/187/5/1113/4604571" target="_blank">here</a>).<br />
<br />
Let's start with allocation concealment. It's critical to successfully randomizing would-be trial participants. When it's done properly, the person adding a participant to a trial has no idea which comparison group that particular person will end up in. So they can't tip the scales out of whack by, say, skipping patients they think wouldn't do well on a treatment, when that treatment is the next slot to allocate.<br />
<br />
Some allocation methods make it easy to succumb to the temptation to crack the system. When allocation is done using sealed envelopes, people <a href="https://www.ncbi.nlm.nih.gov/pubmed/7474192" target="_blank">have admitted</a> to opening the envelopes till they get the one they want - and even going to the radiology department to use a special lamp to see through an opaque envelope, and breaking into a researcher's office to hunt for info! Others have <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.2391" target="_blank">kept logs</a> to try to detect patterns and predict what the next allocation is going to be.<br />
<br />
This happens more often than you might think. A study in 2017 compared sealed envelopes with a system where you have to ring the trial coordinating center to get the allocation. There were 28 clinicians - all surgeons - allocating their patients in this trial. The result:<br />
<blockquote class="tr_bq">
<i>With the sealed envelopes, the randomisation process was corrupted for patients recruited from three clinicians.</i></blockquote>
But there was an overall difference in the ages of people allocated in the whole "sealed envelope" period, too - so some of the others must have peeked now and then, too.<br />
<br />
Messing with allocation was one of the problems that led to a famous trial of the Mediterranean diet being retracted recently. (I wrote about this at <a href="http://blogs.plos.org/absolutely-maybe/2018/06/14/what-does-the-predimed-trial-retraction-re-boot-mean-for-the-mediterranean-diet/" target="_blank">Absolutely Maybe</a> and for the <i><a href="https://blogs.bmj.com/bmj/2018/06/22/hilda-bastian-a-mediterranean-diet-trials-retraction-and-republication-leaves-a-trail-of-questions/" target="_blank">BMJ</a></i>.) Here's what happened, via a report from Gina Kolata (<i><a href="https://www.nytimes.com/2018/06/13/health/mediterranean-diet-heart-disease.html" target="_blank">New York Times</a></i>):<br />
<blockquote class="tr_bq">
<i>A researcher at one of the 11 clinical centers in the trial worked in small villages. Participants there complained that some neighbors were receiving free olive oil, while they got only nuts or inexpensive gifts.</i></blockquote>
<blockquote class="tr_bq">
<i>So the investigator decided to give everyone in the same village the same diet. He never told the leaders of the study what he had done.</i></blockquote>
<blockquote class="tr_bq">
<i>"He did not think it was important".... </i> </blockquote>
But it was: it was obvious on statistical analysis that the groups couldn't have been properly randomized.<br />
<br />
The opportunities to mess up the objectivity of a trial by knowing the allocated group don't end with the randomization. Clinicians could treat people differently, thinking extra care and additional interventions are necessary for people in some groups, or being quicker to encourage people in one group to pull out of the trial. They might be more or less eager to diagnose problems, or judge an outcome measure differently.<br />
<br />
Participants can do the equivalent of all this, too, when they know what group they are in - seek other additional treatments, be more alert to adverse effects, and so on. Ken Schulz lists potential ways clinicians and participants could change the course of a trial <a href="https://pdfs.semanticscholar.org/2198/5326a49a7e6d9d7bc6a938bf032adc4166fb.pdf" target="_blank">here, in Panel 1</a>.<br />
<br />
There's no way of completely preventing bias in a trial, of course. And you can't always blind people to participants' allocation when there's no good placebo, for example. But here are 3 relevant pillars of bias minimization to always look for when you want to judge the reliability of a trial's outcomes:<br />
<br />
<ul>
<li>Adequate concealment of allocation at the front end;</li>
<li>Blinding of participants and others dealing with them during the trial; and</li>
<li>Blinding of outcome assessors - the people measuring or judging outcomes.</li>
</ul>
<br />
Pro tip: Go past the words people use (like "double blind") to see who was being blinded, and what they actually did to try to achieve it. You need to know <i>"Who knew what and when?"</i>, not just what label the researchers put on it.<br />
<br />
<br />
<i>More on blinding <a href="https://statistically-funny.blogspot.com/2012/07/blind-luck-in-praise-of-control-groups.html" target="_blank">here at Statistically Funny</a></i><br />
<br />
<i>6 Tips for Deciphering Outcomes in Health Studies <a href="http://blogs.plos.org/absolutely-maybe/2017/12/21/6-tips-for-deciphering-outcomes-in-health-studies/" target="_blank">at Absolutely Maybe</a>.</i><br />
<br />
<i>Interested in learning more detail about these practices and their history? There's a great essay about the evolution of "allocation concealment" <a href="http://www.jameslindlibrary.org/articles/allocation-concealment-evolution-adoption-methodological-term/" target="_blank">at the James Lind Library</a>.</i><br />
<br />
<br />Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-46108297328169801942017-12-04T07:59:00.002-05:002022-10-02T03:32:25.085-04:00A Science Fortune Cookie<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhY9fby11WsSICbD5nexfEqNXYve-sBWfK6KiFA0Juq3YW-rCNeLy1TSZWj7BE1s2GHDW99l52_Q2LpQ-7xdVPPbPatng2mYm6DLKPSRs6_Po8T0kH1rStIGT6R54XNc0eQ4ri-sDSse9sU/s1600/Effect+size+-+small.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="304" data-original-width="498" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhY9fby11WsSICbD5nexfEqNXYve-sBWfK6KiFA0Juq3YW-rCNeLy1TSZWj7BE1s2GHDW99l52_Q2LpQ-7xdVPPbPatng2mYm6DLKPSRs6_Po8T0kH1rStIGT6R54XNc0eQ4ri-sDSse9sU/s1600/Effect+size+-+small.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
This fortune cookie could start a few scuffles. It's offering a cheerful scenario if you are looking for a benefit of a treatment, for example. But it sure would suck if you are measuring a harm! That's not what's contentious about it, though.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
It's the p values and their size that can get things very heated. The p value is the result you get from a standard test for statistical significance. It can't tell you if a hypothesis is true or not, or rule out coincidence. What it can do is measure an actual result against a theoretical expectation, and let you know if this is pretty much what you would expect to see if a hypothesis is true. The smaller it is, the better: statistical significance is high when the p value is low. Statistical hypothesis testing is all a bit Alice-in-Wonderland!</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
As if it wasn't already complicated enough, people have been dividing rapidly into camps on p values lately. The p value has defenders - we shouldn't dump on the test, just because people misuse it, they say (<a href="http://www.phil.vt.edu/dmayo/personal_website/SENN-Two_Cheers_Paper.pdf">here</a>). Then there are those who think it should be abandoned or at least very heavily demoted (<a href="https://www.nature.com/articles/d41586-017-07522-z">here</a> and <a href="https://www.ncbi.nlm.nih.gov/pubmed/26064558">here</a>, for example).</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Then there is the camp in favor of raising the bar by lowering the level for p values. In September 2017, a bunch of heavy-hitters say the time has come to expect p values to be <i>much</i> tinier, at least when something new is claimed (<a href="https://www.nature.com/articles/s41562-017-0189-z">here</a>).</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
How tiny are they saying a p should be? The usual threshold has been p <0.05 (less than 5%). Instead of that being a significant finding, they decided, just a bit less than 0.05 should only be called "suggestive" of a significant finding. A significant new finding should be way tinier: <0.005.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
That camp reckons support for this change has reached critical mass. Which is suggestive of the <0.05 threshold going the way of the dodo. I have no idea what the fortune cookie on that says! (If you want to read more on avoiding p value potholes, check out <a href="http://blogs.plos.org/absolutely-maybe/2016/04/25/5-tips-for-avoiding-p-value-potholes/">my 5 tips on Absolutely Maybe</a>.)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Now let's get back to the core message of <i>our</i> fortune cookie: the size of a p value is a completely separate issue from the size of the effect. That's because the size of a p value is heavily affected by the size of the study. You can have a highly statistically significant p value for a difference of no real consequence.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There's another trap: an important effect might be real, but the study was too small to know for sure. Here's <a href="https://www.ncbi.nlm.nih.gov/pubmed/28931031">an example</a>. It's a clinical trial of getting people to <a href="https://www.youtube.com/watch?v=hIelhbiuTkc">watch a video</a> about clinical trials, before going through the standard informed consent process to join a hypothetical clinical trial. The control group went through the same consent process, but without the video.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The researchers looked for possible effects on a particular misconception, and on willingness to sign up for a trial. They concluded this (I added the bold):</div>
<div class="separator" style="clear: both;">
<br /></div>
<blockquote class="tr_bq" style="clear: both;">
<i>An enhanced educational intervention augmenting traditional informed consent led to a <b>meaningful reduction</b> in therapeutic misconception <b>without a statistically significant change</b> in willingness to enroll in hypothetical clinical trials.</i></blockquote>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
You need to look carefully when you see statements like this one. You might not be getting an accurate impression. Later, the researchers report:</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
<br /></div>
<blockquote class="tr_bq" style="clear: both;">
<i>[T]his study was powered to detect a difference in therapeutic misconception score but not willingness to participate.</i></blockquote>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
That means they worked out how many people they needed to recruit based only on what was needed to detect a difference of several points in the average misconception scores. Willingness to join a trial dropped by a few percentage points, but the difference wasn't statistically significant. That could mean it doesn't really reduce willingness - or it could mean the study was too small to answer the question. There's just a big question mark: this video reduced misconception, and a reduction in willingness to participate can't be ruled out.</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
What about the effect size? That is how big (or little) the difference between groups is. There are many different ways to measure it. For example, in this trial, "willingness to participate" was simply the proportion of people who said "yes" or "no".</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
However, the difference in "misconception" in that trial was measured by comparing mean results people scored on a test of their understanding. You can brush up on means, and how that leads you to standard deviations and standardized mean differences <a href="https://statistically-funny.blogspot.com/2015/11/more-than-average-confusion-about-what.html" target="_blank">here at Statistically Funny</a>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There are other specific techniques used to set levels of what effect size matters - but those are for another day. In the meantime, there's a technical article explaining important clinical differences <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2716157/">here</a>. And another on <a href="http://staff.bath.ac.uk/pssiw/stats2/page2/page14/page14.html" target="_blank">Cohen's <i>d</i></a>, a measure that is often used in psychological studies. It comes with this rule of thumb: 0.2 is a small effect, 0.5 is medium, and 0.8 is a large effect.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Study reports should allow you to come to your own judgment about whether an effect matters or not. May the next research report you read be written by people who make that easy!</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i>Number needed to confuse: read <a href="https://statistically-funny.blogspot.com/2015/07/arr-or-nnt-whats-your-number-needed-to.html" target="_blank">more at Statistically Funny</a> on the objectivity - or not! - in ways of communicating about effects.</i></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:"MS 明朝";
mso-font-charset:78;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1791491579 18 0 131231 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1107305727 0 0 415 0;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073743103 0 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:Cambria;
mso-ascii-font-family:Cambria;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"MS 明朝";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Cambria;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:Cambria;
mso-ascii-font-family:Cambria;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"MS 明朝";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Cambria;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style><style>
<!--
/* Font Definitions */
@font-face
{font-family:"MS 明朝";
mso-font-charset:78;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1791491579 18 0 131231 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1107305727 0 0 415 0;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073743103 0 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:Cambria;
mso-ascii-font-family:Cambria;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"MS 明朝";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Cambria;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:Cambria;
mso-ascii-font-family:Cambria;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"MS 明朝";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Cambria;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-78245676768899134332016-09-11T15:25:00.004-04:002023-09-14T04:40:39.923-04:00The Highs and Lows of the "Good Study"<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB3yYD85Bkl7ZXCBIeKCDF9RsUezemhjfNBCG_mGwTouOpFmC1WtY7k8bm042UPHkn1zT_2e2IhDLhWjlZLlHAud28P_p2i8-1AkBo4FCDx6wf2tFBqzev39xxoTuQ-Fxj9H7Y56cM-V6omLa6PdUcb5G51LmfxMxF7QDZ7t61Dj9kSYKFFSuoRoeP1LvR/s2037/Weather-study-quality.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Reporting on a study as though it's a weather report. Scientist at a desk has a map behind them, showing "sunny" for education, but cloudy for data on under-18s. (Cartoon by Hilda Bastian.)" border="0" data-original-height="1321" data-original-width="2037" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB3yYD85Bkl7ZXCBIeKCDF9RsUezemhjfNBCG_mGwTouOpFmC1WtY7k8bm042UPHkn1zT_2e2IhDLhWjlZLlHAud28P_p2i8-1AkBo4FCDx6wf2tFBqzev39xxoTuQ-Fxj9H7Y56cM-V6omLa6PdUcb5G51LmfxMxF7QDZ7t61Dj9kSYKFFSuoRoeP1LvR/w400-h260/Weather-study-quality.jpg" width="400" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>
<br />
<br />
Imagine if weather reports only gave the expected average temperature across a whole country. You wouldn't want to be counting on that information when you were packing for a trip to Alaska or Hawaii, would you?<br />
<br />
Yet that's what reports about the strength of scientific results typically do. They will give you some indication of how "good" the whole study is: and leave you with the misleading impression that the "goodness" applies to every result.<br />
<br />
Of course, there are some quality criteria that apply to the whole of a study, and affect everything in it. Say I send out a survey to 100 people and only 20 people fill it in. That low response rate affects the study as a whole.<br />
<br />
You can't just think about the quality of a study, though. You have to think about the quality of each result <i>within</i> that study. The likelihood is, the reliability of data will vary a lot.<br />
<br />
For example, that imaginary survey could find that 25% of people said yes, they ate ice cream every week last month. That's going to be more reliable data than the answer to a question about how many times a week they ate ice cream 10 years ago. And it's likely to be less reliable than their answers to the question, "What year were you born?"<br />
<br />
Then there's the question of missing data. Recently <a href="http://blogs.plos.org/absolutely-maybe/2016/09/02/gender-bias-bias-part-2-unpicking-cherry-picking/">I wrote about</a> bias in studies on the careers of women and men in science. A major data set people often analyze is a survey of people awarded PhDs in the United States. Around 90% of people answer it.<br />
<br />
But within that, the rate of missing data for marital status can be around 10%, while questions on children can go unanswered 4 times as often. Conclusions based on what proportion of people with PhDs in physics are professors will be more reliable than conclusions on how many people with both PhDs in physics and school-age children are professors.<br />
<br />
One of the most misleading areas of all for this are the abstracts and news reports of meta-analyses and systematic reviews. It will often sound really impressive: they'll tell you how many studies, and maybe how many people are in them, too. You could get the impression then, that this means all the results they tell you about have that weight behind them. The <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001419">standard-setting group behind systematic review reporting says</a> you shouldn't do that: you should make it clear with each result. (Disclosure: I was part of that group).<br />
<br />
This is a really big deal. It's unusual for every single study to ask exactly the same questions, and gather exactly the same data, in exactly the same way. And of course that's what you need to be able to pool their answers into a single result. So the results of meta-analyses very often draw on a subset of the studies. It might be a big subset, but it might be tiny.<br />
<br />
To show you the problem, I did a search this morning at the <i>New York Times</i> for "meta-analysis". I picked the first example of a journalist reporting on specific results of a meta-analysis of health studies. <a href="http://well.blogs.nytimes.com/2016/06/27/putting-breast-cancer-on-a-diet/">It's this one</a>: about whether being overweight or obese affects your chances of surviving breast cancer. Here's what the journalist, Roni Caryn Rabin wrote - and it's very typical:<br />
<br />
<i><span style="background-color: white; color: #333333; font-family: "georgia" , "times new roman" , "times" , serif; font-size: 16px; line-height: 23px;"> "Just two years ago, a </span><a href="http://www.ncbi.nlm.nih.gov/pubmed/24769692" style="background-color: white; color: #326891; font-family: georgia, "times new roman", times, serif; font-size: 16px; line-height: 23px;" target="_blank">meta-analysis</a><span style="background-color: white; color: #333333; font-family: "georgia" , "times new roman" , "times" , serif; font-size: 16px; line-height: 23px;"> crunched the numbers from more than 80 studies </span></i><i><span style="background-color: white; color: #333333; font-family: "georgia" , "times new roman" , "times" , serif; font-size: 16px; line-height: 23px;">involving more than 200,000 women with breast cancer, and reported that women who were obese when diagnosed had a 41 percent greater risk of death, while women who were overweight but whose body mass index was under 30 had a 7 percent greater risk".</span></i><br />
<br />
There really was not much of a chance that all the studies had data on that - even though you would be forgiven for thinking that when you looked at <a href="http://www.ncbi.nlm.nih.gov/pubmed/24769692">the abstract</a>. And sure enough, this is how it works out when you <a href="http://annonc.oxfordjournals.org/content/25/10/1901/T1.expansion.html">dig in</a>:<br />
<br />
<ul>
<li>There were 82 studies and the authors ran 31 basic meta-analyses;</li>
<li>The meta-analytic result with the most studies in it included 24 out of the 82;</li>
<li>84% of those results combined 20 or fewer studies - and 58% had 10 or less. Sometimes only 1 or 2 studies had data on a question;</li>
<li>The 2 results the <i>New York Times</i> reported came from about 25% of the studies and less than 20% of the women with breast cancer.</li>
</ul>
<br />
The risk data given in the study's abstract and the <i>New York Times</i> report did not come from "more than 200,000 women with breast cancer". One came from over 42,000 women and the other from over 44,000. In this case, still a lot. Often, it doesn't work that out way, though.<br />
<br />
So be very careful when you think, "this is a good study". That's a big trap. It's not just that all studies aren't equally reliable. The strength and quality of evidence almost always varies <i>within</i> a study.<br />
<br />
<i><br /></i>
<i>Want to read more about this?</i><br />
<br />
<a href="http://clinicalevidence.bmj.com/x/set/static/ebm/learn/665072.html">Here's an overview</a> of the GRADE system for grading the strength of evidence about the effects of health care.<br />
<br />
I've written more about why it's risky to judge a study by its abstract <a href="http://blogs.plos.org/absolutely-maybe/2014/05/12/science-in-the-abstract-dont-judge-a-study-by-its-cover/">at Absolutely Maybe</a>.<br />
<br />
And here's my quick <a href="http://blogs.plos.org/absolutely-maybe/2014/01/20/5-key-things-to-know-about-meta-analysis/">introduction to meta-analysis</a>.<br />
<br />Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com4tag:blogger.com,1999:blog-6353097553819934624.post-71788501635597805542016-08-14T18:41:00.003-04:002022-10-02T03:33:17.096-04:00Cupid's Lesser-Known Arrow<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9oHYWhL_JyppmI5V9bag_Y_Qkdp2J-Yeu5HY60BTUZW7Et2SDFjx5X9B5qCJlDUQF-hg61SkO5UqoEWJtNx10Cm2REmWMmvJ6kcGWh5jsIaWxzvs4SQtZ61a3GekjtBja_SonMEwhKN_S/s1600/Cupid-lesser-known-arrow.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9oHYWhL_JyppmI5V9bag_Y_Qkdp2J-Yeu5HY60BTUZW7Et2SDFjx5X9B5qCJlDUQF-hg61SkO5UqoEWJtNx10Cm2REmWMmvJ6kcGWh5jsIaWxzvs4SQtZ61a3GekjtBja_SonMEwhKN_S/s640/Cupid-lesser-known-arrow.jpg" width="467" /></a></div>
<br />
<br />
<br />
Cupid's famous arrow causes people to fall blindly in love with each other. That can end happily ever after. Not so with his lesser known "immortal time bias" arrow! That one causes researchers to fall blindly in love with profoundly flawed results - and that never ends well.<br />
<br />
This type of time-dependent bias often afflicts observational studies. It's a particular curse for those studies relying on the "big data" from medical records. <a href="http://www.ncbi.nlm.nih.gov/pubmed/26232083"> A recent study</a> found close to 40% of susceptible studies in prominent medical journals were "biased upward by 10% or more". <a href="http://www.ncbi.nlm.nih.gov/pubmed/22342097">A study in 2011</a> found that 62% of studies of postoperative radiotherapy didn't safeguard against immortal time bias. That could make treatment look more effective than it really is.<br />
<br />
So what is it? It's a stretch of time where an outcome couldn't possibly occur for one group - and that gives them a head start over another group. <a href="http://aje.oxfordjournals.org/content/167/4/492.long">Samy Suissa describes a classic case</a> from the early days of heart transplantation in the 1970s. A 1971 study showed 20 people who had heart transplants at Stanford lived an average of 200 days compared to 14 transplant candidates who didn't get them and survived an average of 34 days.<br />
<br />
Those researchers had started the clock from the point at which all 34 people had been accepted into the program. Now of course, all the people who got the transplants were alive at the time of surgery. For the stretch of time they were on the waiting list, they were "immortal": you could not die and still get a heart transplant. So when people on the waiting list died early, they were in the no-transplant group.<br />
<br />
When the data were re-analyzed by others in 1974 to factor this into account, the survival advantage of the operation disappeared. (More about the history in Hanley and Foster's article, <a href="http://ije.oxfordjournals.org/content/43/3/949.long">Avoiding blunders involving 'immortal time'</a>.)<br />
<br />
This bias is also called survivor or survival bias, or survivor treatment selection bias. But time-dependent biases don't only affect death as an outcome. It can affect any outcome, not just death. So "immortal time" isn't really the best term. Hanley and Foster call it event-free time.<br />
<br />
Carl von Walraven and colleagues are among the group that call this kind of phenomenon "competing risk bias":<br />
<br />
<i>Competing risks are events whose occurrence precludes the outcome of interest.</i><br />
<br />
They are the authors of the <a href="http://www.ncbi.nlm.nih.gov/pubmed/26232083">2016 study</a> I mentioned above about how common the problem is. They show the impact on data in a study they did themselves on patient discharge summaries.<br />
<br />
If you were re-admitted to hospital before you got to a physician visit with your discharge summary, you didn't fare as well as the people who went to the doctor. If you just compare the group who went to the physician for follow-up as the hospital encouraged with the group who didn't, the group who didn't visit their doctor had <i>way</i> higher re-admission rates. Not much surprise there, eh?<br />
<br />
Von Walraven says the risk grew as people started to do more time-to-event studies. They put the problem down partly to the popularity of a method for survival ratios that doesn't recognize these risks in its basic analyses. That's Kaplan-Meier risk estimation. You see Kaplan-Meier curves referred to a lot in medical journals.<br />
<br />
Although they're called curves, I think they look more like staircases. Here's an example: number of months survived here starts off the same, but gets better for the blue line after a year, plateauing a couple of years later.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy1EAQOKHD0-Ft2b7WHNmpHzpCy3H5UvuhrG5DIMZ2-DTZqYeJOL0egOx-sn9zL7k2tPfRRDNlxTh0x4pRby7Chq3Z3NDCTz_m82bUQo3_VX3MXumLwfx5VByNhbGhazbg6o5F2SdzXcmS/s1600/Kaplan-Meier.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy1EAQOKHD0-Ft2b7WHNmpHzpCy3H5UvuhrG5DIMZ2-DTZqYeJOL0egOx-sn9zL7k2tPfRRDNlxTh0x4pRby7Chq3Z3NDCTz_m82bUQo3_VX3MXumLwfx5VByNhbGhazbg6o5F2SdzXcmS/s320/Kaplan-Meier.jpg" width="320" /></a></div>
<br />
<span style="font-size: large;"><br /></span>
Some common statistical programs don't have a way to deal with time-dependent calculations in Kaplan-Meier analyses, according to von Walraven. You need extensions of the programs to handle some data properly. The Royal Statistical Society points to this problem too, in the description for their 2-day course on Survival Analysis. (One's coming up in London in <a href="https://events.rss.org.uk/rss/frontend/reg/thome.csp?pageID=4542&eventID=16&traceRedir=2&eventID=16">September 2016</a>.)<br />
<br />
Hanley and Foster have a great guide to recognizing immortal time bias (<a href="http://ije.oxfordjournals.org/content/43/3/949.long">Table 1, page 956</a>). The key, they say, is to "Think person-time, not person":<br />
<br />
<i> If authors used the term 'group', ask... When and how did persons enter a 'group'? Does being in or moving to a group have a time-related requirement?</i><br />
<br />
Given the problem is so common, we have to be <i>very</i> careful when we read <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMHT0025839/">observational studies</a> with time-to-event outcomes and survival analyses. If authors talk about cumulative risk analyses and accounting for time-dependent measures, that's reassuring.<br />
<br />
But what we really need is for the people who do these studies - and all the information gatekeepers, from peer reviewers to journalists - to learn how to dodge this arrow.<br />
<br />
<i>More reading on a somewhat lighter note: my post at <b><a href="http://blogs.plos.org/absolutely-maybe/2016/08/16/winning-or-is-it-the-science-of-winners-fate/">Absolutely Maybe</a></b> on whether winning awards or elections affects longevity.</i><br />
<br />
<br />
~~~~<br />
<br />
<i>The Kaplan-Meier "curve" image was chosen without consideration of its data or the article in which it appears. I used the National Library of Medicine's Open i images database, and erased explanatory details to focus only on the "curve". The source is an article by Kadera BE et al (2013) in </i><a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3750050/">PLOS One</a><i>.</i><br />
<br />
<br />
<br />Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-8969024516786739862015-11-29T10:51:00.005-05:002022-11-05T17:18:22.205-04:00More Than Average Confusion About What Mean Means Mean<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjel1bB8VM-1OjrLV5ToiYAmttX8pP0mz715IfSK2vFL11hB_NT7RVN4LbcaKDVDYZvSs8fZnzNbXAJRPT74rjoYo7qzFR5IwzLGQuooNBa23zrJOo33Wnmks3I1Dgxzow_R3-wuAKALjLTsglZwxQR6G3taE1QlVB5atT9uujPoKsB3MheooDoizzHxQ/s2520/Mean-mediocre-median.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="Cartoon about what people mean when they say average" border="0" data-original-height="1560" data-original-width="2520" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjel1bB8VM-1OjrLV5ToiYAmttX8pP0mz715IfSK2vFL11hB_NT7RVN4LbcaKDVDYZvSs8fZnzNbXAJRPT74rjoYo7qzFR5IwzLGQuooNBa23zrJOo33Wnmks3I1Dgxzow_R3-wuAKALjLTsglZwxQR6G3taE1QlVB5atT9uujPoKsB3MheooDoizzHxQ/w400-h248/Mean-mediocre-median.jpg" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div>
<br />
She's right: on average, when people talk about "average" for a number, they mean the mean.<br />
<br />
The mean is the number we're talking about when we "even out" a bunch of numbers into a single number: 2 + 3 + 4 equals 9. Divide that total by 3 - the number of numbers in that set - and you get the mean: 3.<br />
<br />
But then you hear people make that joke about "almost half the people being below average" - and that's not the mean any more. That's a different average. It's the median - the number in the middle. It comes from the Latin word for "in the middle", just like the word medium. That's why we call the line that runs down the middle of a road the median strip, too.<br />
<br />
If the numbers in a group are all pretty close to each other - like our example here, or, say, the ages of everyone in a class at school - then there's not much difference between the mean and median.<br />
<br />
But if the numbers in a group are wildly far apart - the ages of the people who like Star Wars movies, for example, or whose favorite singer is Frank Sinatra - then it can make a very big difference. Even if <a href="https://www.youtube.com/watch?v=Fd_3EkGr0-4">Strangers In The Night</a> had enough of a resurgence to drag the average age of Ol' Blue Eyes listeners down, the big Sinatra fan base would still skew older!<br />
<br />
How far apart numbers in a dataset are spread from each other is called <a href="https://en.wikipedia.org/wiki/Variance">variance</a>: if the numbers bunch up in the middle, the variance is small. And understanding or dealing with variance is where we start to head in the direction of, well, sort of means of means.<br />
<br />
The distance of a piece of data from the group's mean is a great standard way to measure the spread. This is called the deviation from the mean. A measure called the standard deviation from the mean will be bigger when the numbers are more spread out. Lots of results will cluster within 1 standard deviation (SD), and most will be within 2 standard deviations. Roughly like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7eQVzMRiuxhkKkz99CvcMha2sbfpgv4suwSXLwaci6PBo96SiQdB6nETF8Chr2UKlyHLktglBhoP1EB6ZcObTP9nCXwNxIorvpU2NglWX4xB-93yAh8d61ur34C_iTvUcxLgE2ZiEWfYf/s1600/SD_final_bell_curve.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="176" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7eQVzMRiuxhkKkz99CvcMha2sbfpgv4suwSXLwaci6PBo96SiQdB6nETF8Chr2UKlyHLktglBhoP1EB6ZcObTP9nCXwNxIorvpU2NglWX4xB-93yAh8d61ur34C_iTvUcxLgE2ZiEWfYf/s320/SD_final_bell_curve.jpg" width="320" /></a></div>
<br />
<br />
From here, it's a hop, skip to another calculation based on the mean that you often come across in health studies. It's a way to standardize the differences in means (average results) called the standardized mean difference (SMD).<br />
<br />
The SMD needs to be used when outcomes have been measured in similar, but different, ways in groups that researchers are comparing.<br />
<br />
There's a lot you can make sense of when you know what the means mean!<div><br /></div><div>
<br />
<br />
<i>The SMD is calculated by dividing the differences in the mean in two groups by standard deviations. You can read more on standard deviations <a href="http://statistically-funny.blogspot.com/2013/04/dont-worry-its-just-standard-deviation.html">here at Statistically Funny</a>.</i><br />
<br />
<i>Feel like testing your knowledge of the mean, median, and mode? (The mode is the number in a set that occurs the most often: so if our example had been 2 + 3 + 4 + 4, then the mode would have been 4.) Try the <a href="https://www.khanacademy.org/math/probability/descriptive-statistics/central_tendency/e/mean_median_and_mode">Khan Academy quiz</a>.</i><br />
<i><br /></i><i>Interested in the ancient roots of averages? Examples from Herodotus, Thucydides, and in Homer <a href="https://web.archive.org/web/20160303185417/https://www.amstat.org/publications/jse/v11n1/bakker.html">here</a> (very academic).</i></div><div><i><br /></i>
<i><br /></i></div><div><i>Note: Edited to address broken links, on November 6, 2022.</i></div><div><i><br /></i></div><div><i>Hilda Bastian</i></div><div><i><br /></i></div>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com2tag:blogger.com,1999:blog-6353097553819934624.post-75821954356503466932015-09-30T07:42:00.001-04:002015-09-30T14:47:38.837-04:00AGHAST! The Day the Trial Terminator Arrived<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVC4TV8M-P1yAHZsGC23WvBWs2v183OHf9IoVaDP61CmUacxFJhunb7EQxn9Xl_Z9roAEnpQNU3ESnu4QQx5kSIHyxIigX-e7R2dBn1drx5S6KQDpU-JabxgjpZf4Ptijb7SEQgAicbVrt/s1600/Data-Monitoring-Police-small.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVC4TV8M-P1yAHZsGC23WvBWs2v183OHf9IoVaDP61CmUacxFJhunb7EQxn9Xl_Z9roAEnpQNU3ESnu4QQx5kSIHyxIigX-e7R2dBn1drx5S6KQDpU-JabxgjpZf4Ptijb7SEQgAicbVrt/s1600/Data-Monitoring-Police-small.jpg" /></a></div>
<br />
<br />
Clinical trials are complicated enough when everything goes pretty much as expected. When it doesn't, the dilemma of continuing or stopping can be excruciatingly difficult. Some of the greatest dramas in clinical research are going on behind the scenes around this. Even who gets to call the shot can be bitterly disputed.<br />
<br />
A trial starts with a plan for how many people have to be recruited to get an answer to the study's questions. This is calculated based on what's known about the chances of benefits and harms, and how to measure them.<br />
<br />
Often a lot is known about all of this. Take a trial of antibiotics, for example. How many people will end up with gastrointestinal upsets is fairly predictable. But often the picture is so sketchy it's not much more than a stab in the dark.<br />
<br />
Not being sure of the answers to the study's questions is an <a href="http://www.ncbi.nlm.nih.gov/pubmed/8707498">ethical prerequisite</a> for doing clinical trials. That's called equipoise. The term was coined by lawyer <a href="https://en.wikipedia.org/wiki/Charles_Fried">Charles Fried</a> in his 1974 book, <a href="https://books.google.com/books/about/Medical_Experimentation.html?id=xWj2AAAACAAJ" style="font-style: italic;">Medical Experimentation</a>. He argued that the investigator should be uncertain if people were going to be randomized. <a href="http://www.ncbi.nlm.nih.gov/pubmed/3600702">In 1987</a>, Benjamin Freedman argued the case for clinical equipoise: that we need professional uncertainty, not necessarily individual uncertainty.<br />
<br />
It's hard enough to agree if there's uncertainty at any time! But the ground can shift gradually, or even dramatically, while a trial is chugging along.<br />
<br />
I think it's helpful to think of this in 2 ways: a shift in knowledge caused by the experience in the trial, and external reasons.<br />
<br />
Internal issues that can put the continuation of the trial in question include:<br />
<ul>
<li>Not being able to recruit enough people to participate (by far the most common reason);</li>
<li>More serious and/or frequent harm than expected tips the balance;</li>
<li>Benefits much greater than expected;</li>
<li>The trial turns out to be futile: the differences in outcome between groups is so small, even if the trial runs its course, we'll be none the wiser (<a href="http://onlinelibrary.wiley.com/doi/10.1002/sim.2151/abstract">PDF</a>).</li>
</ul>
<div>
External developments that throw things up in the air or put the cat among the pigeons include:</div>
<div>
<ul>
<li>A new study or other data about benefits or safety - especially if it's from another similar trial;</li>
<li>Pressure from groups who don't believe the trial is justified or ethical;</li>
<li><a href="http://www.ncbi.nlm.nih.gov/pubmed/15994362">Commercial reasons</a> - a manufacturer is pulling the plug on developing the product it's trialing, or just can't afford the trial's upkeep;</li>
<li>Opportunity costs for public research sponsors <a href="http://www.ncbi.nlm.nih.gov/pubmed/19513106">has been argued</a> as a reason to pull the plug for possible futility, too.</li>
</ul>
</div>
Sometimes several of those things happen at once. Stories about several examples are in a companion post to this one over at <a href="http://blogs.plos.org/absolutely-maybe/2015/09/30/the-mess-that-trials-stopped-early-can-leave-behind/">Absolutely Maybe</a>. They show just how difficult these decisions are - and the mess that stopping a trial can leave behind.<br />
<br />
Trials that involve the risk of harm to participants should have a plan for monitoring the progress of the trial without jeopardizing the trial's integrity. Blinding or masking the people assessing outcomes and running the trial is a key part of trial methodology (more about that <a href="http://blogs.plos.org/absolutely-maybe/2015/08/31/5-key-things-to-know-about-data-on-adverse-effects/">here</a>). Messing with that, or <a href="http://statistically-funny.blogspot.com/2013/01/data-bingo-oh-no.html">dipping into the data often</a>, could end up leading everyone astray. Establishing stopping rules before the trial begins is the safeguard used against that - along with a committee of people other than the trial's investigators monitoring interim results.<br />
<br />
Although they're called stopping "rules", they're actually more guideline than rule. And other than having it done independently of the investigators, there is no one widely agreed way to do it - including the role of the sponsors and their access to interim data.<br />
<br />
Some methods focus on choosing a one-size-fits-all threshold for the data in the study, while others are more Bayesian - taking external data into account. There is a detailed look at this in a <a href="http://www.journalslibrary.nihr.ac.uk/hta/volume-9/issue-7#hometab0">2005 systematic review</a> of trial data monitoring processes by Adrian Grant and colleagues for the UK's National Institute of Health Research (NIHR). They concluded there is no strong evidence that the data should stay blinded for the data monitoring committee.<br />
<br />
A <a href="http://www.ncbi.nlm.nih.gov/pubmed/16684642">2006 analysis</a> HIV/AIDS trials stopped early because of harm, found that only 1 out of 10 had established a rule for this before the trial began but it's more common these days. A <a href="http://www.ncbi.nlm.nih.gov/pubmed/20332404">2010 review</a> of trials stopped early because the benefits were greater than expected found that 70% mentioned a data monitoring committee (DMC). (These can also be called data and safety monitoring boards (DSMBs) or data monitoring and ethics committees (DMECs).)<br />
<br />
Despite my cartoon of data monitoring police, DMCs are only advisors to the people running the trial. They're not responsible for the interpretation of a trial's results, and what they do generally remains confidential. Who other than the DMC gets to see interim data, and when, is a debate that can get very heated.<br />
<br />
Clinical trials only started to become common <a href="http://www.ncbi.nlm.nih.gov/pubmed/20877712">in the 1970s</a>. <a href="http://www.ncbi.nlm.nih.gov/pubmed/24908333">Richard Stephens writes</a> that it was only in the 1980s, though, that keeping trial results confidential while the trial is underway became the expected practice. In some circumstances, Stephens and his colleagues argue, publicly releasing interim results while the trial is still going on can be a good idea. They talk about examples where the release of interim results saved trials that would have foundered because of lack of recruitment from clinicians who didn't believe the trial was necessary.<br />
<br />
One approach when there's not enough knowledge to make reliable trial design decisions is a type of trial called an <a href="http://www.ncbi.nlm.nih.gov/pubmed/22917111">adaptive trial</a>. It's designed to run in steps, based on what's learned. About 1 in 4 might adapt the trial in some way (<a href="http://bcb.dfci.harvard.edu/calendar/Zelen/slides/1SPocock_Some%20Current%20Controversies%20in%20Clinical%20Trials%20Research.pdf">PDF</a>). It's relatively early days for those.<br />
<br />
In the end, no matter which processes are used, weighing up the interests of the people in the trial, with the interests of everyone else in the future who could benefit from more data, will be irreducibly tough. <a href="http://www.ncbi.nlm.nih.gov/pubmed/17577008">Steven Goodman writes</a> that we need more people with enough understanding and experience of the statistics and dilemmas involved in data monitoring committees.<br />
<br />
We also need to know more about when and how to bring people participating in the trial into the loop - including having community representation on DMCs. Informing participants at key points more would means some will leave. But most might stay, as they did in the Women's Health Initiative hormone therapy trials (<a href="http://kooperberg.fhcrc.org/papers/2007anderson.pdf">PDF</a>) and one of the <a href="http://www.ncbi.nlm.nih.gov/pubmed/8095008">AZT trials</a> in the earlier years of the HIV epidemic.<br />
<br />
There is one clearcut issue here. And that's the need to release the results of any trial when it's over, regardless of how or why it ended. That's a clear ethical obligation to the people who participated in the trial - the <a href="http://www.ncbi.nlm.nih.gov/pubmed/23360313">desire to advance knowledge and help others</a> is one of the reasons many people agree to participate. (More on this at the <a href="http://www.alltrials.net/find-out-more/all-trials/">All Trials campaign</a>.)<br />
<br />
<br />
<i>More at Absolutely Maybe: </i><a href="http://blogs.plos.org/absolutely-maybe/2015/09/30/the-mess-that-trials-stopped-early-can-leave-behind/">The Mess That Trials Stopped Early Can Leave Behind</a><br />
<i><br /></i>
~~~~<br />
<i><br /></i>
<i>Trial acronyms: If someone really did try to make an artificial gallbladder - not to mention actually start a trial on it! - I think lots of us would be pretty aghast! But a lot of us are pretty aghast about the mania for trial acronyms too. More on that <a href="http://statistically-funny.blogspot.com/2012/06/trial-acronymania-menace.html">here at Statistically Funny</a>.</i><br />
<i><br /></i>
<i><br /></i>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-74727309160528571762015-07-19T16:28:00.003-04:002015-07-25T10:56:45.227-04:00ARR OR NNT? What's Your Number Needed To Confuse?<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrLhfXEkagM1E_ZkNnqqXea9I2S8STTd26k_QiXQQ8_XKrgG8s6AOGe_XruYevlM8nJ9i4tXuXlENFklH7GvCYaPpD_AB1HwpjrUD3kpLBQauTMWfvBkXM5iSIG5GDYc9KZu1yYM3IFA4y/s1600/Number-needed-to-confuse.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="366" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrLhfXEkagM1E_ZkNnqqXea9I2S8STTd26k_QiXQQ8_XKrgG8s6AOGe_XruYevlM8nJ9i4tXuXlENFklH7GvCYaPpD_AB1HwpjrUD3kpLBQauTMWfvBkXM5iSIG5GDYc9KZu1yYM3IFA4y/s400/Number-needed-to-confuse.jpg" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
I used to think numbers are completely objective. Words, on the other hand, can clearly stretch out, or squeeze, people's perceptions of size. "OMG that spider is <b><i>HUGE</i></b>!" "Where? What - <i><b>that</b></i> little thing?"<br />
<br />
Yes, numbers can be more objective than words. Take adverse effects of health care: if you use the word "common" or "rare", people won't get <a href="http://www.ncbi.nlm.nih.gov/pubmed/25155972">as accurate an impression</a> as if you use numbers.<br />
<br />
But that doesn't mean numbers are completely objective. Or even that numbers are always better than words. Numbers get a bit elastic in our minds, too.<br />
<br />
We're mostly good at sizing up the kinds of quantities that we encounter in real life. For example, it's pretty easy to imagine a group of 20 people going to the movies. We can conceive pretty clearly what it means if 18 say they were on the edge of the seats the whole time.<br />
<br />
There's an evolutionary theory about this, called ecological rationality. The idea is, our ability to reason with quantities developed in response to the quantities around us that we frequently need to mentally process. (More on this in Brase [<a href="http://www.kstate.co/psych/research/documents/2002JBDM.pdf">PDF</a>] and Gigerenzer and Hoffman [<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.3201&rep=rep1&type=pdf">PDF</a>].)<br />
<br />
Whatever the reason, we're just not as good at calibrating risks that are lower frequency (Yamagishi [<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.414.6919&rep=rep1&type=pdf">PDF</a>]). We're going to get our heads around 18 out 20 well. But 18000 out of 200000? Not so much. We'll do pretty well at 1 out of 10, or 1 out of 100 though.<br />
<br />
And big time trouble starts if we're reading something where the denominators are jumping around - either toggling from percent to per thousand and back, or saying "7 out of 13 thought the movie was great, while 4 out of 19 thought it was too scary, and 9 out of 17 wished they had gone to another movie". We'll come back to this in a minute. But first, let's talk about some key statistics used to communicate the effects of health care.<br />
<br />
Statistics - where words and numbers combine to create a fresh sort of hell!<br />
<br />
First there's the problem of the elasticity in the way our minds process the statistics. That means that whether they realize it or not, communicators' choice of statistic can be manipulative. Then there's the confusion created when people communicate statistics with words that get the statistics wrong.<br />
<br />
Let's look at some common measures of effect sizes: absolute risk (AR), relative risk (RR), odds ratio (OR), and number needed to treat (NNT). (The evidence I draw on is summarized <a href="http://blogs.plos.org/absolutely-maybe/2015/03/09/mind-your-ps-rrs-and-nnts-on-good-statistics-behavior/">in my long post here</a>.)<br />
<br />
Natural frequencies are the easiest thing for people generally to understand. And getting more practice with natural frequencies might help us to get better at reasoning with numbers, too (Gigerenzer again [<a href="http://umg.umdnj.edu/smdm/pdf/16-03-273.pdf">PDF</a>]).<br />
<br />
Take our movie-goers again. Say that 6 of the 20 were hyped-up before the movie even started. And 18 were hyped-up afterwards. Those are natural frequencies. If I give you those "before and after" numbers in percentages, that's "absolute risk" (AR). Lots of people (but not everybody) can manage the standardization of percentages well.<br />
<br />
But if I use relative risks (RR) - people were 3 times as likely to be hyped-up after seeing that movie - then the all-important context of proportion is lost. That's going to sound like a lot, whether it's a tiny difference or a huge difference. People will often react to that without stopping to check, "yes, but from what to what?" From 6 to 18 out of 20 is a big difference. But going from 1 out of a gazillion to 3 out of a gazillion just ain't much worth crowing or worrying about.<br />
<br />
RRs are critically important: they're needed for calculating a personalized risk if you're not at the same risk as the people in a study, for example. But if it's the only number you look at, you can get an exaggerated idea.<br />
<br />
So sticking with absolute risks or natural frequencies, and making sure the baseline is clear (the "before" number), is better at helping people understand an effect. Then they can put their own values on it.<br />
<br />
The number needed to treat, takes the change in absolute change and turns it upside down. (Instead of calculating the difference out of 100, it's 100 divided by the difference.) So that instead of the constant denominator of 100, you now have ones that change: instead of 60% of people being hyped-up because of the movie, it becomes NNT 1.7 (1.7 people have to see the movie for 1 person to get hyped-up).<br />
<br />
This can be great in some circumstances, and many people are really used to NNTs. But on average, this is one of the hardest effect measures to understand. Which means that it's easier to be manipulated by it.<br />
<br />
NNT is the anti-RR if you like: RRs exaggerate, NNTs minimize. Both can mislead - and that can be unintentional or deliberate.<br />
<br />
When it comes to communicating with people who need to use results, I think using only statistics that will frequently mislead because it's a preference of the communicator is paternalistic, because it denies people the right to an impression based on their own values. Like all forms of paternalism, that's sometimes justified. But there's a problem when it becomes the norm.<br />
<br />
The NNT was developed in the 1990s [<a href="http://ebm.bmj.com/content/1/6/164.full.pdf">PDF</a>]. It was meant to do a few things - including counteracting the exaggeration of the RR. Turns out it overshot the mark there! It was also intended to be easier to understand than the odds ratio (OR).<br />
<br />
The OR brings us to the crux of the language problems. People use words like odds, risks, and chances interchangeably. Aaarrrggghhh!<br />
<br />
A risk in statistics is what we think of as our chances of being in the group: a 60% absolute risk means a 60 in 100 (or 6 in 10) "chance".<br />
<br />
An odds ratio in statistics is like odds in horse-racing and other gambling. It factors in both the odds of "winning" versus the odds of "losing". (If you want to really get your head around this, check out <i><a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0050876/">Know Your Chances</a> </i>by Woloshin, Schwartz, and Welch. It's a book that's been <a href="http://www.ncbi.nlm.nih.gov/pubmed/17310049">shown in trials</a> to work!)<br />
<br />
The odds ratio is a complicated thing to understand, especially if it's embedded in confusing language. It's a very sound way to deal with data from some types of studies, though. So you see odds ratios a lot in <a href="http://blogs.plos.org/absolutely-maybe/2014/01/20/5-key-things-to-know-about-meta-analysis/">meta-analyses</a>. (If you're stumped about getting a sense of proportion in a meta-analysis, look at the number of events and the number of participants - they are the natural frequencies.)<br />
<br />
There's one problem that all of these ways of portraying risks/chances have in common: when people start putting them in sentences, they frequently get the language wrong. So they can end up communicating something entirely other than what was intended. You really need to double-check exactly what the number is, if you want to protect yourself from getting the wrong impression.<br />
<br />
OK, then, so what about "pictures" to portray numbers? Can that get us past the problems of words and numbers? Graphs, smile-y versus frown-y faces, and the like? Many think this is "the" answer. <i>But</i>...<br />
<br />
This is going to be useful in some circumstances, misleading in others. <a href="http://www.ncbi.nlm.nih.gov/pubmed/14512488">Gerd Gigerenzer and Adrian Edwards</a>: "Pictorial representations of risk are not immune to manipulation either". (A topic for another time, although I deal with it a little in the "5 shortcuts" post listed below.)<br />
<br />
Where does all this leave us? Few researchers reporting data have the time to invest in keeping up with the literature on communicating numbers - so while we can plug away at improving the quality of reporting of statistics, there's no overnight solution there.<br />
<br />
Getting the hang of the common statistics yourself is one way. But the two most useful all-purpose strategies could involve detecting bias.<br />
<br />
One is to sharpen your skills at detecting people's ideological biases and use of spin. Be on full alert when you can see someone is utterly convinced and trying to persuade you with all their chips on a particular way of looking at data - especially if it's data on a single outcome. If the question matters to you, <a href="http://statistically-funny.blogspot.com/2013/11/does-it-work-beware-of-too-simple-answer.html">beware of the too-simple answer</a>.<br />
<br />
The second? Be on full alert when you see something you really want, or don't want, to believe. The biggest bias we have to deal with is our own.<br />
<br />
<br />
<i>More of my posts relevant to this theme:</i><br />
<i><br /></i>
<i><a href="http://statistically-funny.blogspot.com/2013/11/does-it-work-beware-of-too-simple-answer.html">Does It Work? Beware of the Too-Simple Answer</a></i><br />
<i><br /></i>
<i>At Absolutely Maybe (PLOS Blogs):</i><br />
<i><a href="http://blogs.plos.org/absolutely-maybe/2014/10/13/5-shortcuts-to-keep-data-on-risks-in-perspective/">5 Shortcuts to Keep Data on Risks in Perspective</a></i><br />
<i><a href="http://blogs.plos.org/absolutely-maybe/2015/03/09/mind-your-ps-rrs-and-nnts-on-good-statistics-behavior/">Mind your "p"s, RRs, and NNTs: On Good Statistics Behavior</a></i><br />
<i><br /></i>
<i>At Third Opinion (MedPage Today):</i><br />
<i><a href="http://www.medpagetoday.com/Blogs/ThirdOpinion/52540">The Trouble With Evidence-Based Medicine, the 'Brand'</a></i><br />
<i><a href="http://www.medpagetoday.com/Blogs/ThirdOpinion/50273">The NNT: An Overhyped and Confusing Statistic</a></i><br />
<i><br /></i>
Check out <b><a href="http://hildabastian.net/">hildabastian.net</a> </b>for a running summary of what I'm writing about.<br />
<br />
<br />Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com4tag:blogger.com,1999:blog-6353097553819934624.post-25200618038612198102015-02-08T12:17:00.002-05:002022-10-02T03:33:35.719-04:00Let's Play Outcome Mash-up - A Clinical Trial Shortcut Classic!<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjovengJBHD52ccr6Noih1SkuJ7dexwqKpXUThyZEfRVf9rOOZSqhMlVhh783Sfd5P-xywkcasXISvNDZ9s43Bv2x-xvEQa1u96cGNb_RX1zsiDkPhf2xAEwyvTi3RSWLJSPm17nIrIVx4/s1600/Composite-outcomes.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjovengJBHD52ccr6Noih1SkuJ7dexwqKpXUThyZEfRVf9rOOZSqhMlVhh783Sfd5P-xywkcasXISvNDZ9s43Bv2x-xvEQa1u96cGNb_RX1zsiDkPhf2xAEwyvTi3RSWLJSPm17nIrIVx4/s1600/Composite-outcomes.jpg" width="386" /></a></div>
<br />
<br />
Deciphering trial outcomes can be a tricky business. As if many measures aren't hard enough to make sense of on their own, they are often combined in a complex maneuver called a composite endpoint (CEP) or composite outcome. The composite is treated as a single outcome. And journalists often phrase these outcomes in ways that give the impression that each of the separate components has improved.<br />
<br />
Here's an example from the <a href="http://www.nytimes.com/2014/11/18/health/study-finds-alternative-to-statins-in-preventing-heart-attacks-and-strokes.html?_r=0" target="_blank">New York Times</a>, reporting on the results of a major trial from the last American Heart Association conference:<br />
<blockquote class="tr_bq">
"There were 6.4% fewer cardiac events - heart disease deaths, heart attacks, strokes, bypass surgeries, stent insertions and hospitalization for severe chest pain..."</blockquote>
That individual statement sounds like the drug reduced deaths, bypasses, stents, and hospitalization for unstable angina, doesn't it? But it didn't. The modest effect was on non-fatal heart attacks and stroke only.*<br />
<br />CEPs are increasingly common: by 2007, <a href="http://www.ncbi.nlm.nih.gov/pubmed/18981486" target="_blank">well over a third </a>of cardiovascular trials were using them. CEPs are a clinical trial shortcut because you need fewer people and less time to hit a jackpot. A trial's main pile of chips is riding on its pre-specified <a href="http://www.esourceresearch.org/eSourceBook/ClinicalTrials/4Endpoints/tabid/200/Default.aspx" target="_blank">primary outcome</a>: the one that answers the trial's central, most important question.<br />
<br />
The primary outcome determines the size and length of the trial, too. For example, if the most important outcome for a chronic disease treatment is to increase the length of people's lives, you would need a lot of people to get enough events to count (the event in this case would be death). And it would take years to get enough of those events to see if there's anything other than a dramatic, sudden difference.<br />
<br />
But if you combine it with one or more other outcomes - like non-fatal heart attacks and strokes - you'll get enough events much more quickly. Put in lots, and you're really hedging your bets.<br />
<br />It's a very valuable statistical technique - but it can go haywire. Say you have 3 very serious outcomes that happen about as often as each other - but then you add another component that is less serious and much more common. The number of less serious events can swamp the others. Everything could even be riding on only one less serious component. But the CEP has a very impressive name - like "serious cardiac events." Appearances can be deceptive.<br />
<br />
Enough data on the nature of the events in a CEP should be clearly reported so that this is obvious, <a href="http://www.ncbi.nlm.nih.gov/pubmed/17403713" target="_blank">but it often isn't</a>. And even if the component events are reported deep in the study's detail, don't be surprised if it's not pointed out in the abstract, press release, and publicity!<br />
<br />
There are several different ways a composite can be constructed, including use of techniques like weighting that need to be transparent. Because it's combining events, there has to be a way of dealing with what happens when more than one event happens to one person - and that's not always done the same way. The definitions might make it obvious, the most serious event might count first according to a hierarchy, or the one that happened to a person first might be counted. But exactly what's happening often won't be clear - maybe even <a href="http://www.ncbi.nlm.nih.gov/pubmed/18981486" target="_blank">most of the time</a>.<br />
<br />
There's agreement on some things you should look out for (see for example <a href="http://www.ncbi.nlm.nih.gov/pubmed/15761002" target="_blank">Montori</a>, <a href="http://www.ncbi.nlm.nih.gov/pubmed/17537898" target="_blank">Hilden</a>, and <a href="http://www.ncbi.nlm.nih.gov/pubmed/25632312" target="_blank">Rauch</a>). Are each of the components as serious as each other and/or likely to increase (or decrease) together in much the same way? If one's getting worse and one's getting better, this isn't really measuring one impact.<br />
<br />
The biggest worry, though, is when researchers play the slot machine in my cartoon (what we call <a href="https://en.wikipedia.org/wiki/Slot_machine" target="_blank">the pokies</a>, "Downunder"). I've stressed the dangers of hunting over and over for a statistical association (<a href="http://statistically-funny.blogspot.com/2014/03/if-at-first-you-dont-succeed.html" target="_blank">here</a> and <a href="http://statistically-funny.blogspot.com/2013/01/data-bingo-oh-no.html" target="_blank">here</a>). The <a href="http://www.ncbi.nlm.nih.gov/pubmed/18981486" target="_blank">analysis by Lim and colleagues</a> found some suggestion that component outcomes are sometimes selected to rig the outcome. If it wasn't the pre-specified primary outcome, and it wasn't specified in the original entry for it in a <a href="https://clinicaltrials.gov/ct2/info/about" target="_blank">trials register</a>, that's a worry. Then it wasn't really a tested hypothesis - it's a new hypothesis.<br />
<br />
Composite endpoints, properly constructed, reported, and interpreted are essential to getting us decent answers to many questions about treatments. Combining death with serious non-fatal events makes it clear when there's a drop in an outcome largely because people died before that could happen, for example. But you have to be very careful once so much is compacted into one little data blob.<br />
<br />
<br />
<i> </i><i>(<a href="http://my.americanheart.org/idc/groups/ahamah-public/@wcm/@sop/@scon/documents/downloadable/ucm_469669.pdf" target="_blank">Check out slide 14</a> to see the forest plot of results for the individual components the journalist was reporting on. <a href="http://statistically-funny.blogspot.com/2012/10/the-forest-plot-trilogy-gripping.html" target="_blank">Forest plots are explained here</a> at Statistically Funny.)</i><br />
<i><br /></i>
<i><b><br /></b></i><br />
<i><b>More on understanding clinical trial outcomes:</b></i><br />
<br />
<ul>
<li><i>Another way to get clinical trial results quickly: <a href="http://statistically-funny.blogspot.com/2014/11/biomarkers-unlimited-accept-only-our.html" target="_blank">surrogates and biomarkers</a> (at Statistically Funny)</i></li>
<li><i><a href="http://blogs.plos.org/absolutely-maybe/5-shortcuts-to-keep-data-on-risks-in-perspective/" target="_blank">Keeping risks in perspective </a>(at Absolutely Maybe)</i></li>
</ul>
<br />
<i><br /></i>
<i><b>New this week: </b>I'm delighted to now have a third blog, one for physicians with the wonderful team at <a href="http://www.medpagetoday.com/" target="_blank">MedPage Today</a>. It's called <a href="http://www.medpagetoday.com/Blogs/ThirdOpinion/" target="_blank">Third Opinion</a>.</i><br />
<i><br /></i>
<i><br /></i>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com1tag:blogger.com,1999:blog-6353097553819934624.post-83862485247338904812014-11-30T23:24:00.003-05:002022-10-02T03:35:02.134-04:00Biomarkers Unlimited: Accept Only OUR Substitutes!<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJVjtyT7c8z9IsGqixOVr3z5YA3U-POAhu2sF4XNyo0byFVVFt5_o8Dhfy8u2wtZ9EL248a-elh9G_mKosMbkOaR-5P1AmOVWvKM7VIHYkxGa4i5a9rjeHj5t1kPXDwQmdsi6RBmc6qACc/s1600/Biomarkers-Unlimited.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJVjtyT7c8z9IsGqixOVr3z5YA3U-POAhu2sF4XNyo0byFVVFt5_o8Dhfy8u2wtZ9EL248a-elh9G_mKosMbkOaR-5P1AmOVWvKM7VIHYkxGa4i5a9rjeHj5t1kPXDwQmdsi6RBmc6qACc/s1600/Biomarkers-Unlimited.jpg" width="393" /></a></div>
<br />
Sounds great, doesn't it? Getting clinical trial results quickly has so much going for it. Information sooner! More affordable trials!<br />
<br />
Substituting outcomes that can take years, or even decades, to emerge, with ones you can measure much earlier, makes clinical research much simpler. This kind of substitute outcome is called a surrogate (or intermediate) endpoint or outcome.<br />
<br />
Surrogates are often biomarkers - biological signs of disease or a risk factor of disease, like cholesterol in the blood. They are used in clinical care to test for, or keep track of, signs of emerging or progressing disease. Sometimes, like cholesterol, they're the target of treatment.<br />
<br />
The problem is, these kinds of substitute measures aren't always reliable. And sometimes we find that out in the hardest possible way.<br />
<br />
The risk was recognized as soon as the current methodology of clinical trials was being developed in the 1950s. <b><a href="https://en.wikipedia.org/wiki/Austin_Bradford_Hill" target="_blank">Austin Bradford Hill</a></b>, who played a leading role, put it bluntly: if the "rate falls, the pulse is steady, and the blood pressure impeccable, we are still not much better off if unfortunately the patient dies."<br />
<br />
<b><a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0050887/#ch2.s2" target="_blank">That famously happened</a></b> with some drugs that controlled cardiac arrhythmia - irregular heartbeat that increases the chances of having a heart attack. On the basis of ECG tests that showed the heartbeat was regular, these drugs were prescribed for years before a trial showed that they were causing tens of thousands of premature deaths, not preventing them. That kind of problem has <b><a href="http://www.ncbi.nlm.nih.gov/pubmed/15863552" target="_blank">happened too often</a></b> for comfort.<br />
<br />
It happened again this week - although at least before the drug was ever approved. A drug company <b><a href="http://investors.amgen.com/phoenix.zhtml?c=61656&p=irol-newsArticle&ID=1992492" target="_blank">canceled all its trials</a></b> for advanced gastric (stomach) cancer of a new drug. The drug is called <b><a href="http://www.ncbi.nlm.nih.gov/pubmed/?term=Rilotumumab" target="_blank">Rilotumumab</a></b>. <b><a href="http://www.onclive.com/publications/obtn/2013/December-2013/Treatment-Advances-Bring-New-Hope-in-Gastric-Cancer" target="_blank">Back in January</a></b>, it was a "promising" treatment, billed as bringing "new hope in gastric cancer." It got through the early testing phases and was in <b><a href="http://www.nlm.nih.gov/services/ctphases.html" target="_blank">Phase III trials</a></b> - the kind needed to get FDA approval.<br />
<br />
But one phase III trial, <b><a href="http://clinicaltrials.gov/ct2/show/NCT01697072?term=rilomet-1&rank=1" target="_blank">RILOMET-1</a></b>, quickly showed an increase in the number of deaths in people using the drug. We don't know how many yet - but it was enough for the company to decide to end all trials of the substance.<br />
<br />
This drug targets a biomarker associated with worse disease outcomes, an area seen by some as <b><a href="http://www.ncbi.nlm.nih.gov/pubmed/25142842" target="_blank">transforming</a></b> gastric cancer research and treatment. Others see <b><a href="http://www.ncbi.nlm.nih.gov/pubmed/24487788" target="_blank">considerable challenges</a></b>, though - and what happened to the participants in the RILOMET-1 trial underscores why.<br />
<br />
There is a lot of controversy about surrogate outcomes - and debates about what's needed to show that an outcome or measure is a <b><a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3287814/" target="_blank">valid surrogate</a> </b>we can rely on. They can lead us to think that a treatment is <b><a href="http://www.ncbi.nlm.nih.gov/pubmed/23360719" target="_blank">more effective</a></b> than it really is.<br />
<br />
Yet <b><a href="http://www.medpagetoday.com/PublicHealthPolicy/FDAGeneral/48244" target="_blank">a recent investigative report</a></b> found that cancer drugs are being increasingly approved based only on surrogate outcomes, like "progression-free survival." That measures biomarker activity rather than overall survival (when people died).<br />
<br />
It can be hard to recognize at first, what's a surrogate and what's an actual health outcome. One rule of thumb is, if you need a laboratory test of some kind, it's more likely to be a surrogate. Whereas symptoms of the disease you're concerned, or harm caused by the disease, are the direct outcomes of interest. Sometimes those are specified as"patient-relevant outcomes."<br />
<br />
Many surrogate outcomes are incredibly important, of course - <b><a href="http://www.thelancet.com/journals/laninf/article/PIIS1473-3099%2814%2970896-5/abstract" target="_blank">viral load</a></b> for HIV treatment and trials for example. But in general, when clinical research results are based only on surrogates, the <b><a href="http://www.ncbi.nlm.nih.gov/pubmed/21802903" target="_blank">evidence just isn't as strong and reliable</a></b> as it is for the outcomes we are really concerned about.<br />
<br />
<i>~~~~</i><br />
<i><br /></i>
<i><b><a href="http://statistically-funny.blogspot.com/2012/06/promising-over-hyped-under-tested.html" target="_blank">See also, Statistically Funny on "promising" treatments.</a></b></i><br />
<i><br /></i>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com1tag:blogger.com,1999:blog-6353097553819934624.post-43839695424746997922014-10-12T02:01:00.000-04:002015-12-05T09:19:32.289-05:00Sheesh - what are those humans thinking?<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEF5g0xfxu7r_m0kGCmZNJK_w7zlwoqXJXJKO151px0SS0s5mtk8jHWxP83lMjf4HZ4Mc7lAUqt2FNlUH8mWVgSwVKk77ZSZesO5F6vOgkuaA3_sF51Yvwaq-paYpLFnpE-Z3OaIwvr3CE/s1600/Cecil-sheesh.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEF5g0xfxu7r_m0kGCmZNJK_w7zlwoqXJXJKO151px0SS0s5mtk8jHWxP83lMjf4HZ4Mc7lAUqt2FNlUH8mWVgSwVKk77ZSZesO5F6vOgkuaA3_sF51Yvwaq-paYpLFnpE-Z3OaIwvr3CE/s640/Cecil-sheesh.jpg" width="497" /></a></div>
<div style="text-align: center;">
<br /></div>
<br />
I can neither confirm nor deny that Cecil is now a participant in one of the there-is-no-limit-to-the-human-lifespan resveratrol studies at <a href="https://www.yahoo.com/health/researcher-hunting-the-key-to-aging-believes-95193101362.html">Harvard's "strictly guarded mouse lab"</a>! If he is, I'm sure he's even more baffled by the humans' hype over there.<br />
<br />
Resveratrol is the antioxidant in grapes that many believe makes drinking red wine healthy. And it's a good example of how research on animals is often terribly misleading and misinterpreted. I've written about it over at <a href="http://blogs.plos.org/absolutely-maybe/2014/08/04/resveratrol-hangover-waking-up-after-hypothesis-bingeing/">Absolutely Maybe</a> if you're interested in a classic example of the rise and fall of animal-research-based hype (or more detail about resveratrol).<br />
<br />
But this week, it's media hype about a study using human stem cells in mice in another lab at Harvard that's made me ratty. You could get the idea that a human trial of a "cure" for type 1 diabetes is just a matter of time now - and not a lot of time at that. According to the <a href="http://news.harvard.edu/gazette/story/2014/10/giant-leap-against-diabetes/" target="_blank">leader of the team</a>, Doug Melton, "We are now just one preclinical step away from the finish line."<br />
<br />
An effective treatment that ends the need for insulin injections would be incredibly exciting. But we see this kind of claim from laboratory research all the time, don't we? How often does it work out - even for the studies that <b>are</b> at "the finishing line" for animal studies?<br />
<br />
Not all that often: maybe <a href="http://www.ncbi.nlm.nih.gov/pubmed/17032985" target="_blank">about a third of the time</a>.<br />
<br />
<a href="http://www.ncbi.nlm.nih.gov/pubmed/20361020" target="_blank">Bart van der Worp</a> and colleagues wrote an excellent paper explaining why. It's not just that other animals are so different from humans. We're far less likely to hear of the failed animal results than we are of human trials that don't work out as hoped. That bias towards positive published results draws an over-optimistic picture.<br />
<br />
As well as fundamental differences between species, van der Worp points to other common issues that reduce the applicability for humans of typical studies in other animals:<br />
<br />
<ul>
<li>The animals tend to be younger and healthier than the humans who have the health problem;</li>
<li>They tend to be a small group of animals that are very similar to each other, while the humans with the problem are a large very varied group;</li>
<li>Only male or only female animals are often used; and</li>
<li>Doses higher than humans will be able to tolerate are generally used.</li>
</ul>
Limited <a href="http://www.ncbi.nlm.nih.gov/pubmed/25486354" target="_blank">genetic diversity</a> could be an issue, too.<br />
<br />
So how does the Harvard study fare on that score? They used stem cells to develop insulin-producing cells that appeared to function normally when transplanted into mice. But this was the very early stages. When it came to the test they reported on the ones with diabetes, there were only 6 (young) mice who got the transplants (and 1 died) (plus a comparison group). Gender was not reported - and as is common in laboratory animal studies, there wasn't lengthy follow-up. This was an important milestone, but there's a very long way to go here. Transplants in humans face <a href="http://www.ipscell.com/2014/10/top-10-takeaways-from-harvard-stem-cell-diabetes-paper/" target="_blank">a lot of obstacles</a>.<br />
<br />
Van der Worp points to another set of problems: inadequacies in research methods that we've learned over time in human research bias the proceedings too much - including problems with statistical analyses. <a href="http://www.ncbi.nlm.nih.gov/pubmed/24906117" target="_blank">Jennifer Hirst</a> and colleagues have studied this too. They concluded that so many studies were bedeviled by issues such as lack of randomization and blinding by those assessing outcomes, that they should never have been regarded as being "the finishing line" before human experimentation at all.<br />
<br />
There's good news though! <a href="http://www.dcn.ed.ac.uk/camarades/" target="_blank">CAMARADES</a> is working to improve this - with the same approach for chipping away at these problems as in human trials: by slogging away at biased methodologies and publication bias. And pushing for good quality systematic reviews of animal studies before human trials are undertaken. It's well worth half an hour to watch <a href="https://www.youtube.com/watch?v=g9HJbmwcYBk">the wonderful talk</a> by Emily Sena at Evidence Live 2015.<br />
<br />
Laboratory animal research may be called "preclinical," but even that jargon is a bit of over-optimistic marketing. Most of what's tried in the lab will never get near human trials. And when it does, it will mostly be disappointing. Laboratory research is needed, and encouraging progress is great. But people should definitely not be getting our hopes up too much about it.<i><br /></i><br />
<i><br /></i>
<i>~~~~</i><br />
<i><br /></i>
<i>The National Institutes of Health (NIH) addressed the issue of <a href="http://www.nature.com/news/policy-nih-to-balance-sex-in-cell-and-animal-studies-1.15195" target="_blank">gender in animal experiments</a> earlier in 2014. After I wrote this post, the NIH also released proposed <a href="http://www.nih.gov/about/reporting-preclinical-research.htm" target="_blank">guidelines for reporting preclinical research</a>.</i><br />
<i><br /></i>
<i>Thanks to <a href="https://twitter.com/phylogenomics" target="_blank">Jonathan Eisen</a> for adding a link for the full text of the paper to <a href="http://www.ncbi.nlm.nih.gov/pubmed/25303535#cm25303535_6583" style="font-weight: bold;" target="_blank">PubMed Commons</a>, as well as to a blog post by <a href="https://twitter.com/pknoepfler" target="_blank">Paul Knoepfler</a> discussing the context of the stem cell work by Felicia Pagliuca, Doug Melton and colleagues. NHS Behind the Headlines have also <a href="http://www.nhs.uk/news/2014/10October/Pages/Cure-for-type-1-diabetes-within-reach.aspx" target="_blank">analyzed and explained this study</a>.</i><br />
<i><br /></i>
<i>Thanks to <a href="https://twitter.com/JimJohnsonSci" target="_blank">Jim Johnson</a> for pointing an oversight: that animal studies - this one included - can also suffer from having too little follow-up.</i><br />
<i><br /></i>
<i>Interest declaration: I'm an academic editor at one of the journals whose papers on animal research I commended (PLOS Medicine) and on the human ethics advisory group of another (PLOS One), but I had no involvement in either paper.</i><br />
<div style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; font-size: 13px; line-height: 19px;">
<br />
Update: Checked, post and cartoon refreshed, and link to Sena's talk at Evidence Live on 5 December 2015.</div>
Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0tag:blogger.com,1999:blog-6353097553819934624.post-85780800621921348292014-03-16T14:32:00.002-04:002019-05-19T03:05:57.622-04:00If at first you don't succeed...<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpghqz2QllgaNtdO9vlQlqH2cYjhDsCMMXYW8Ci3PzhDWqNie-PbysAiBFazAimLL8ExMq0jU511LlFQbGV8ZvvD2b96u0uclgIB1Tbkd99Nje7QPvFvm_ReQCGLEfZGel6_T5rDXU_ubW/s1600/Significus.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpghqz2QllgaNtdO9vlQlqH2cYjhDsCMMXYW8Ci3PzhDWqNie-PbysAiBFazAimLL8ExMq0jU511LlFQbGV8ZvvD2b96u0uclgIB1Tbkd99Nje7QPvFvm_ReQCGLEfZGel6_T5rDXU_ubW/s640/Significus.jpg" width="492" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
If only post hoc analyses <u>always</u> brought out the inner skeptic in us all! Or came with red flashing lights instead of just a little token "caution" sentence buried somewhere. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Post hoc analysis is when researchers go looking for patterns in data. (Post hoc is Latin for "after this.") Testing for <a href="http://statistically-funny.blogspot.com/2013/03/nervously-approaching-significance.html" target="_blank">statistically significant</a> associations is not by itself a way to sort out the true from the false. (More about that <a href="http://blogs.plos.org/absolutely-maybe/2016/04/25/5-tips-for-avoiding-p-value-potholes/">here</a>.) Still, many treat it as though it is - especially when they haven't been able to find a "significant" association, and turn to the bathwater to look for unexpected babies.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Even when researchers know the scientific rules and limitations, funny things happen along the way to a final research report. It's the problem of <a href="http://www.ncbi.nlm.nih.gov/pubmed/22006061" target="_blank">researchers' degrees of freedom</a>: there's a lot of opportunity for picking and choosing, and changing horses mid-race. Researchers can succumb to the temptation of over-interpreting the value of what they're analyzing, with <a href="http://www.ncbi.nlm.nih.gov/pubmed/22006061" target="_blank">"convincing self-justification."</a> (See the <a href="https://via.hypothes.is/https://europepmc.org/abstract/MED/24417410#annotations:M9b1NH47EeioSBN95O_8tg" target="_blank">moving goalposts over time here</a>, for example, as trialists are faced with results that didn't quite match their original expectations.)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
And even if the researchers don't read too much into their own data, someone else will. That interpretation can quickly turn a statistical artifact into a "fact" for many people.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Let's look more closely at Significus' pet hate: post hoc analyses. There are dangers inherent in <a href="http://statistically-funny.blogspot.com/2013/01/data-bingo-oh-no.html" target="_blank">multiple testing</a> when you don't have solid reasons for looking for a specific association. The more often you randomly dip into data without a well-founded target, the higher your chances of pulling out a result that will later prove to be a dud.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both;">
It's a little like fishing in a pond where there are random old shoes among the fish. The more often you throw your fishing line into the water, the greater your chances of snagging a shoe instead of a fish.</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
<a href="http://sys.mahec.net/media/onlinejournal/moore_thongs_flipflops.pdf" target="_blank">Here's a study</a> designed to show this risk. The data tossed up significant associations such as: women were more likely to have a cesarean section if they preferred butter over margarine, or blue over black ink.</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
The problem is huge in areas where there's a lot of data to fish around in. For published genome-wide association studies, for example, over <a href="http://www.ncbi.nlm.nih.gov/pubmed/11882781" target="_blank">90% of the "associations"</a> with a disease couldn't consistently be found again. Often, researchers don't report <a href="http://www.ncbi.nlm.nih.gov/pubmed/12727138" target="_blank">how many tests were run</a> before they found their "significant" results, which makes it impossible for others to know how big a problem multiple testing might be in their work.</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
The problem extends to subgroup analyses where there is not an established foundation for an association. The <a href="http://www.ncbi.nlm.nih.gov/pubmed/22422832" target="_blank">credibility of claims</a> made on subgroups in trials is low. And it has serious consequences. For example, an early trial suggested only men with stroke-like symptoms benefit from aspirin - which <a href="http://www.ncbi.nlm.nih.gov/pubmed/24449319" target="_blank">stopped many doctors</a> from prescribing aspirin to women.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
How should you interpret post hoc and subgroup analyses then? If analyses were not pre-specified <b>and</b> based on established, plausible reasons for an association, then <a href="http://statistically-funny.blogspot.com/2013/04/look-ma-straight-as.html" target="_blank">one study isn't enough to be sure</a>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
With subgroups that weren't randomized as different arms of a trial, it's <a href="http://www.ncbi.nlm.nih.gov/pubmed/24449319" target="_blank">not enough that the average</a> for one subgroup is higher than the average for another subgroup. There could be other factors influencing the outcome other than their membership of that subgroup. An <a href="https://en.wikipedia.org/wiki/Interaction_(statistics)" target="_blank">interaction test</a> is done to try to account for that.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
It's more complicated when it's a meta-analysis, because there are so many differences between one study and another. The exception here is an <a href="http://www.ctu.mrc.ac.uk/research_areas/meta-analysis/individual_patient_data.aspx" target="_blank">individual patient data meta-analysis</a>, which can study differences between patients directly.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
In the end, it comes down to being careful not to see a new hypothesis generated by research as a "fact" already proven by the study from which it came.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Post hoc, ergo propter hoc. This description of <a href="https://en.wikipedia.org/wiki/Post_hoc_ergo_propter_hoc" target="_blank">basic faulty logic</a> - "after this, therefore <b>because</b> of this" - is <a href="http://books.google.com/books?id=eK0a5CgyV7kC&pg=PA56&lpg=PA56&dq=aristotle+post+hoc+ergo+propter+hoc&source=bl&ots=Psz_Magz5p&sig=iDNMZXhf5i9__nlawsKe8_AqhL0&hl=en&sa=X&ei=MBMhU_qwNObI0QGbzIHwBQ&ved=0CEwQ6AEwAw#v=onepage&q=aristotle%20post%20hoc%20ergo%20propter%20hoc&f=false" target="_blank">as ancient as the language</a> that made it famous. We've had millennia to snap out of the dangerous mental shortcut of seeing a cause where there's only coincidence. Yet we still hurtle like lemmings over cliffs into its alluring clutches.</div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: center;">
<i><a href="http://statistically-funny.blogspot.com/2013/01/data-bingo-oh-no.html" target="_blank">More on multiple testing at Statistically Funny.</a></i></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="color: #0000ee;"><i><u><br /></u></i></span><i><a href="http://statistically-funny.blogspot.com/2013/01/data-bingo-oh-no.html" target="_blank"></a></i></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJVkvxR3D1JGqgxXsyM_Js9txJmLJ-1C4jLzyx-bqW_D45rlYKf3S5aDVgfgcxUHclGFSskOn1HSj0uhVihBCarISvHjntwL8-0eVZg32MLgxXBZAi2PaksmbPfkKNaZB7hL0T2yOm6X8B/s1600/Eureka.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="260" data-original-width="258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJVkvxR3D1JGqgxXsyM_Js9txJmLJ-1C4jLzyx-bqW_D45rlYKf3S5aDVgfgcxUHclGFSskOn1HSj0uhVihBCarISvHjntwL8-0eVZg32MLgxXBZAi2PaksmbPfkKNaZB7hL0T2yOm6X8B/s1600/Eureka.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<i></i></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com4tag:blogger.com,1999:blog-6353097553819934624.post-77707285623449357572013-12-29T16:05:00.001-05:002022-10-09T22:18:00.578-04:00What's so good about "early," anyway?<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjw3m6lAijsJDUD1NBlioR_5ZB9Kx62Jc2gJbLUnc0HihgVeLzs2DxlOBJ7F3M0_JDFeWKl5VUSvoLvttGRRAZmW-5REA7UrIPAVi_tmHy5z_6_fAOcTkvtTBPTFHMFTDWrZ0aI7tbvle-E/s1600/Gertrud.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="366" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjw3m6lAijsJDUD1NBlioR_5ZB9Kx62Jc2gJbLUnc0HihgVeLzs2DxlOBJ7F3M0_JDFeWKl5VUSvoLvttGRRAZmW-5REA7UrIPAVi_tmHy5z_6_fAOcTkvtTBPTFHMFTDWrZ0aI7tbvle-E/s400/Gertrud.jpg" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
"Early." It's one of those words like "new" and "fast," isn't it? As though they are inherently good, and their opposites - "late," "old" and "slow" - are somehow bad.<br />
<br />
Believing in the value and virtue of being an early bird has deep roots in our cultural consciousness. It goes back at least as far as ancient Athens. Aristotle's <a href="http://books.google.com/books?id=D-4kAAAAMAAJ&printsec=frontcover&dq=aristotle+politics+and+economics&hl=en&sa=X&ei=7ZG_UvipCZOjsQTlroDoDg&ved=0CC8Q6AEwAA#v=onepage&q=aristotle%20politics%20and%20economics&f=false">treatise on household economics</a> said that early rising was both virtuous and beneficial: "It is likewise well to rise before daybreak; for this contributes to health, wealth and wisdom."<br />
<br />
But just as Gertrud came to suspect the benefits for her of being early weren't all they were cracked up to be, earliness isn't always better in other areas either. The "get in early!" assumption has an in-built tendency to lead us astray when it comes to detection of diseases and conditions. And <a href="http://www.ncbi.nlm.nih.gov/pubmed/22393129">even most physicians</a> - just the <a href="http://statistically-funny.blogspot.com/2013/01/fright-night-in-doctors-lounge.html">people we often rely on</a> to inform us - don't understand enough about the pitfalls that lead us to jump to conclusions about early detection too, well…early.<br />
<br />
<b>Pitfall number 1:</b> Those who need it least get the most early detection<br />
<br />
This one is a double-edged sword. Firstly, whether it's a screening program or research studying early detection, there tends to be a "worried well" or "healthy volunteer" effect (selection bias). It's easy to have higher than average rates of good health outcomes in people who are at low risk of bad ones anyway. This can lead to inflated perceptions of how much benefit is possible.<br />
<br />
The other problem is an <a href="http://cebp.aacrjournals.org/content/16/5/998.full.pdf+html">over-supply of fatalism</a> among many people who may be able to materially benefit from early detection. Constant bombardment about all the things they could possibly be worrying about <a href="http://www.ncbi.nlm.nih.gov/pubmed/23790111">might even make it more likely</a> that they <a href="http://www.ncbi.nlm.nih.gov/pubmed/18708374">shut out vital information</a> - which could make it even more likely that they ignore symptoms, for example.<br />
<br />
<b>Pitfall number 2: </b>Over-diagnosis from detecting people who would never have become ill from the condition detected<br />
<br />
This one is called length bias. For many conditions, like cancers, there are dangerous ones that develop too quickly for a screening program to catch them. Early detection is actually better at detecting ones that may never threaten their health. More people die with cancer, than of it.<br />
<br />
So early detection means many people are fighting heroic battles that were never necessary. And some will actually be harmed by parts of some screening processes that carry serious risks of their own (like colonoscopies), or adverse effects of the treatments they got which they didn't need.<br />
<br />
Add to those the number of people who are diagnosed as being "at risk" of conditions they will never have or which would have resolved without treatment, and the number harmed is depressingly huge.<br />
<br />
This massive swelling of the numbers of people who have survived phantoms is spreading the shadow of angst ever wider (a subject I've written about in relation to cancer at <a href="http://blogs.scientificamerican.com/guest-blog/2012/10/18/nihmim12-the-spreading-shadow-of-cancer-angst-3-things-you-need-to-know-to-meet-it-rationally/">Scientific American</a>). Spend 10 minutes or so listening to Iona Heath on this subject - <a href="http://www.youtube.com/watch?v=oFh1kJ7GCGQ">starting just past 2 minutes on this video</a>. [And read @Deevybee's important comment and links about developmental conditions in early childhood below.]<br />
<br />
<b>Pitfall number 3: </b>The statistical effect that means survival rates "improve" even if no one's life expectancy increases<br />
<br />
This is lead-time bias. And it's why you should always be careful when you see survival rates in connection with early detection and treatment. Screening programs, by definition, are for people who have no symptoms (pre-clinical). So they cut the part of your life where you don't know you have the disease short. Even if the earlier diagnosis made no difference to the length of your life, the amount of time you lived with knowledge of the disease (disease "survival") is longer.<br />
<br />
What we want is to move the needle on length and/or quality of life. For that to happen, there has to be safe and effective treatment, safe and effective screening procedures, and more people found at a time they can be helped than would have come from diagnosing the condition when there symptoms.<br />
<br />
Here's an example. This person's disease began when they were 40 years old. They lived without any problem from it until 76 years old - then they died when they were 80. Their disease survival was less than 5 years. The proportion of their life that they "had" the disease was short.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpJKL3LMDwK46JZWu0xDLiZ_JlY919Uczzpx6T5ErEh0iftY-GS_V7dBn4rXXk4TgmcThSM9IqRXCuttZE7GHN5tYs8PJxQKgdOa6dZakWeMsshRBRMuj8ghBFJGgPJYCMR84IKKEs001H/s1600/Lead-time-symptoms.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpJKL3LMDwK46JZWu0xDLiZ_JlY919Uczzpx6T5ErEh0iftY-GS_V7dBn4rXXk4TgmcThSM9IqRXCuttZE7GHN5tYs8PJxQKgdOa6dZakWeMsshRBRMuj8ghBFJGgPJYCMR84IKKEs001H/s400/Lead-time-symptoms.jpg" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
Now here's the same person, with early detection that made no difference to when they died - but the needle on how long they have "had" the disease has shifted. So they now survive longer than 5 years with the disease. The "lead time" has changed, but survival in the way we mean it hasn't changed at all.<br />
<i><br /></i>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0h1obwj8OKRWKAL3VA7reCwoq9Y3OUmDezbZdHt2pWEl5s1KnE5x05boB8FSZzaXVLeuzdWekeVN4tgjh8y37b52371qOM6Y_juFzrQ1iswGjTxqnNYtEAiIRGfU3fMCUmMfCgC1dzAtt/s1600/Lead-time-screening.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0h1obwj8OKRWKAL3VA7reCwoq9Y3OUmDezbZdHt2pWEl5s1KnE5x05boB8FSZzaXVLeuzdWekeVN4tgjh8y37b52371qOM6Y_juFzrQ1iswGjTxqnNYtEAiIRGfU3fMCUmMfCgC1dzAtt/s400/Lead-time-screening.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Randomized trials are needed to establish that in fact early detection and intervention programs do more good than harm - some do, some don't.</div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i>More Statistically Funny on screening - <a href="http://statistically-funny.blogspot.com/2012/10/you-have-right-to-remain-anxious.html">"You have the right to remain anxious"</a> and on over-diagnosis: <a href="http://statistically-funny.blogspot.com/2012/04/over-abundance-of-over-diagnosis.html">here</a> and <a href="http://statistically-funny.blogspot.com/2012/12/the-diagnosing-disorders-epidemic.html">here</a>.</i></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i>Here's a fact sheet about <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0033150/">what you need to know about screening tests</a>. And <a href="http://www.acponline.org/clinical_information/journals_publications/ecp/marapr99/primer.htm">here's a little more technical primer</a> of the 3 biases explained here.</i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i>Or my post at PLOS Blogs, <a href="http://blogs.plos.org/absolutely-maybe/the-disease-prevention-illusion-a-tragedy-in-five-parts/">The Disease Prevention Illusion: A Tragedy in Five Parts</a>.</i></div>
<i><br /></i>
Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com6tag:blogger.com,1999:blog-6353097553819934624.post-8318302774334447132013-11-17T13:55:00.001-05:002017-02-11T08:12:32.090-05:00Does it work? Beware of the too-simple answer<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYJMwqeNAyXu2R8yuxA4g-x_cxmPzMkWOBsPyXgvfdXQVgS6ZGTBlSfi8nWWnyhUvPhjztOh8iQNQY9ahinuocackEeTSZDv4pXo-4ymMhI06oj35accgiWACzrfLrkCa5ctWPJo6d_yhr/s1600/Does_it_work.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYJMwqeNAyXu2R8yuxA4g-x_cxmPzMkWOBsPyXgvfdXQVgS6ZGTBlSfi8nWWnyhUvPhjztOh8iQNQY9ahinuocackEeTSZDv4pXo-4ymMhI06oj35accgiWACzrfLrkCa5ctWPJo6d_yhr/s1600/Does_it_work.jpg" width="281" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Leonard is so lucky! He's just asked a very complicated question and he's not getting an over-confident and misleading answer. Granted, he was likely hoping for an easier one! But let's dive into it.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
"Does": that verb packs a punch. How <b>do</b> we know whether something does or doesn't work? It would be great if that were simple, but unfortunately it's not.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
I talk a lot here at Statistically Funny about the need for trials and systematic reviews of them to help us find the answers to these questions. But whether we're talking about trials or other forms of research, statistical techniques are needed to help make sense of what emerges from a study.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Too often, this aspect of research is going to lead us down a garden path. It's common for people to take the approach of relying only, or largely, on testing for <a href="http://statistically-funny.blogspot.com/2013/03/nervously-approaching-significance.html">statistical significance</a>. People often assume this means it's categorical proof of whether or not something was a coincidence.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
However a statistically significant result - especially from a single study - is often misunderstood and contributes to over-confidence about what we know. It's not a magical wand that finds out the truth. The numbers alone cannot possibly "know" that. Here's my quick overview: <a href="http://blogs.plos.org/absolutely-maybe/2016/04/25/5-tips-for-avoiding-p-value-potholes/">5 tips</a> to avoid getting this wrong. I wrote about testing for statistical significance in some detail at <a href="http://blogs.plos.org/absolutely-maybe/statistical-significance-and-its-part-in-science-downfalls/" target="_blank">Absolutely Maybe</a>. Leonard's statistician is a <a href="https://en.wikipedia.org/wiki/Bayesian_probability">Bayesian</a>: you can find out some more about that, too, in that post.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
As chance would have it, there was also a lot of discussion this week in response to <a href="http://www.pnas.org/content/early/2013/10/28/1313476110.full.pdf+html?with-ds=yes">a paper published</a> while I was writing that post. It called for a tightening of the threshold for significance, which isn't really the answer either. Thomas Lumley puts that into great perspective over at his wonderful blog, <a href="http://notstatschat.tumblr.com/post/67132955516/moving-the-goalposts">Biased and Inefficient</a>: a very valuable read.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
"It": now that part of our question should be easy, right? Actually, this can be particularly tricky. The treatment you could be using may not be very much like the one that was studied. Even if it's a prescription drug, the dose or regimen you're facing might not be the same as the one used in studies. Or it might be used with another intervention that could affect how it works.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Then there's the question of whether "it" is even what it says it is. Unlike prescription drugs, for example, the contents of herbal remedies and dietary supplements aren't closely regulated <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0004991/">to ensure that what it says on the label is what's inside</a>. That was also recently in the news, and <a href="http://www.forbes.com/sites/emilywillingham/2013/11/07/study-herbal-supplements-full-of-contaminants-substitutes-and-fillers/">covered in detail here by Emily Willingham</a>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
If it's a non-drug intervention, it's actually highly likely that the articles and other reports of the research don't ever make clear exactly what "it" is. Paul Glasziou had a brainwave about this: he's started <a href="http://www.racgp.org.au/your-practice/guidelines/handi/">HANDI: the Handbook of Non-Drug Intervention</a>. When a systematic reviews shows that something works, the HANDI team wants to dig out all the details and make sure we all know exactly what "it" is.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
For example, if you heard that drinking water before meals can help you lose weight, and you want to try it, HANDI helpfully points out what that actually means is drinking half a liter of water before every meal AND having a low-calorie diet.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
"Work": this one really needs to get specific. You really need to be thinking about each possible outcome separately - the evidence is going to vary in quality and quantity from one outcome to another. I explain this in another post <a href="http://statistically-funny.blogspot.com/2016/09/the-highs-and-lows-of-good-study.html">here</a>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Think of it this way: if you do a survey with 150 questions in it, there are going to be more answers to some of the questions than others. For example, if you had 400 survey respondents, they might all have answered the first easy question and there could be virtually no answers to a hard question near the end. So thinking "a survey of 400 people found…" an answer to that later question is going to be seriously misleading.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
On top of that, you have to take the <a href="http://statistically-funny.blogspot.com/2013/07/alleged-effects-include-howling.html">possible adverse effects</a> into account, too. There can be complicated trade-offs between effects. And how much does it work for a particular outcome? Does a sliver of a benefit count to you as "working"? That might be enough for the person answering your question, but it might not be enough for it to count for you - especially if there are risks, costs or inconvenience involved.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
And <b>who</b> did it work for in the research? Whether or not research results apply to a person in your situation can be straightforward, but it might not be.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Then there's the question of how high did researchers set the bar? Did the treatment effect have to be superior to doing nothing, or doing something else - or is the information coming from comparing it to something else that itself may not be all that effective? You might think that can't possibly happen, but it does more often than you might think. You can find out about this here at Statistically Funny, where I tackle the issue of drugs that are <a href="http://statistically-funny.blogspot.com/2012/08/drugs-go-head-to-head-at-pharma-olympics.html">"no worse (more or less)."</a> </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Finally, one of the most common trip-ups of all: did they really measure the outcome, or a proxy for it? If it's a proxy for the real thing, how good a proxy is it? The use of s<a href="https://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0072533/">urrogate measures or biomarkers</a> is increasing fast: more about that <a href="http://statistically-funny.blogspot.com/2014/11/biomarkers-unlimited-accept-only-our.html">in this post</a>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
So while there are many who might have told Leonard, "Yes, it's been proven to work in clinical trials" in a few seconds flat, I wonder how long it would take his statistician to answer the question? There are no stupid questions, but beware of the too-simple answer.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com7tag:blogger.com,1999:blog-6353097553819934624.post-84879753504488659482013-09-16T03:22:00.001-04:002022-10-02T03:49:29.923-04:00More than one kind of self-control<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKWfdn-I4XZ0fTJiYGRP9R6Fg_bfPJMZZek1NzXng8pO-c3GTIlXpxCI2PGZYK_zyiYUqXJoC-1tshLr4DVpuOjIl_KUCfaB3P-0Hy6rf5_5y-MjHXglodvFOO4V4r_R40yqMNgKy7lcJU/s1600/Twilling-sisters.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="362" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKWfdn-I4XZ0fTJiYGRP9R6Fg_bfPJMZZek1NzXng8pO-c3GTIlXpxCI2PGZYK_zyiYUqXJoC-1tshLr4DVpuOjIl_KUCfaB3P-0Hy6rf5_5y-MjHXglodvFOO4V4r_R40yqMNgKy7lcJU/s400/Twilling-sisters.jpg" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
If you like reading randomized trials about skin and oral health treatments - and who doesn't? - you come across a few split-face and split-mouth ones. Instead of randomizing groups of people to different interventions so that a group of people can be a control group (parallel trials), sections of a person are randomized.<br />
<br />
It's not only done with faces and teeth. Pairs of body parts can be randomized too, like arms or legs. These studies are sometimes called "within-person" trials. This kind of randomization means that you <a href="http://www.ncbi.nlm.nih.gov/pubmed/17716311">need fewer people in the trial</a>, because you don't have to account for all the variations between human beings.<br />
<br />
It has to be a treatment that affects only the specific area of the body treated, though. Anything that could have an influence on the "control" part is called a <a href="http://www.ncbi.nlm.nih.gov/pubmed/2201705">spill-over effect</a>. There are still inevitably things that happen that affect the whole person, and those have to be accounted for with this kind of trial. Body part randomization is one of several ways a person can be their own control: the <a href="http://statistically-funny.blogspot.com/2012/11/the-one-about-ship-wrecked.html">n of 1 trial</a> is another way.<br />
<br />
Randomizing sections didn't start in trials with people: it began with <a href="http://jmp.info/software/pdf/30612.pdf">split-plot experiments</a> in agricultural research. The idea was developed by the pioneer statistician, <a href="http://www-history.mcs.st-andrews.ac.uk/Biographies/Fisher.html">Sir Ronald Aylmer Fisher</a>, who had done breeding experiments. He explained the technique in his classic 1925 text, <a href="http://trove.nla.gov.au/work/10809098">"Statistical Methods for Research Workers."</a><br />
<br />
It's great to see that neither blackheads nor treatment effects are hampering the Twilling sisters' style! They do seem to be at risk of susceptibility to the skincare industry's hard sells, though. Those issues are the subject of my post <a href="http://blogs.plos.org/absolutely-maybe/2013/09/16/blemish-the-truth-about-blackheads/">Blemish: The Truth About Blackheads</a>.<br />
<i><br /></i>
<i><br /></i>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com1tag:blogger.com,1999:blog-6353097553819934624.post-63773281470326729882013-07-28T13:55:00.003-04:002022-10-02T03:35:27.995-04:00Alleged effects include howling<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgHRU6w-ce3aCxnAfJ5EXyQ5QFQLeNn2Lsv_LmlI4ye6LMXKyyxMw_9acEei_i1fGK5HGcViNZJJTwgqo5f5VbuHiiaeJpRV9kW3qKQFfn0AvMVWWk4uJXT_gayAF4BDHEkXg1YvQWJ2BO/s1600/Cause-and-effect.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgHRU6w-ce3aCxnAfJ5EXyQ5QFQLeNn2Lsv_LmlI4ye6LMXKyyxMw_9acEei_i1fGK5HGcViNZJJTwgqo5f5VbuHiiaeJpRV9kW3qKQFfn0AvMVWWk4uJXT_gayAF4BDHEkXg1YvQWJ2BO/s640/Cause-and-effect.jpg" width="547" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
When dogs howl at night, it's <a href="http://scienceline.ucsb.edu/getkey.php?key=340">not the full moon</a> that sets them off. <a href="http://thebark.com/content/dog-speak-sounds-dogs">Dogs are communicating</a> for all sorts of reasons. We're just not all that good at understanding what they're saying.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
We make so many mistakes about attributing cause and effect <a href="http://en.wikipedia.org/wiki/List_of_cognitive_biases">for so many reasons</a>, that it's almost surprising we get it right as often as we do. But all those mistaken beliefs we realize we have, don't seem to teach us a lesson. <span class="Apple-style-span" style="font-style: normal;">Pretty soon after catching ourselves out, we're at it again, taking mental shortcuts, being <a href="http://en.wikipedia.org/wiki/Cognitive_miser">cognitive misers</a>.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
It's so pervasive, you would think we would know this about ourselves, at least, even if we don't understand dogs. Yet we commonly under-estimate how much bias is affecting our beliefs. That's been dubbed the <a href="http://en.wikipedia.org/wiki/Bias_blind_spot">bias blind spot</a> that we tend to live in.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Even taking all that into account, "effect" is an astonishingly over-used word, especially in research and science communication where you would hope people would be more careful. The maxim that correlation (happening at the same time) does not necessarily mean causation has <a href="http://www.slate.com/articles/health_and_science/science/2012/10/correlation_does_not_imply_causation_how_the_internet_fell_in_love_with_a_stats_class_clich_.html">spread far and wide, becoming something of a cliche</a> along the way.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
But does that mean that people are as careful with the use of the word "effect" as they are with the use of the "cause" word? Unfortunately not.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Take this common one: "Side effects include...." Well, actually, don't be so fast to swallow that one. Sometimes, genuine adverse effects will follow that phrase. But more often, the catalogue that follows is not adverse effects, but a list of adverse <b>events</b> - things that happened (or were reported). Some of them may be causally related, some might not be.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
You have to look carefully at claims of benefits and harms. Even researchers who aren't particularly biased will word it carelessly. You will often hear that 14% experienced nausea, say - without it being pointed out that 13% of people on placebos also experienced nausea, and the difference wasn't <a href="http://statistically-funny.blogspot.com/2013/03/nervously-approaching-significance.html">statistically significant</a>. Some adverse effects are well known, and it doesn't matter (diarrhea and antibiotics, say). That's not always so, though. (More on this: <a href="http://blogs.plos.org/absolutely-maybe/2015/08/31/5-key-things-to-know-about-data-on-adverse-effects/">5 Key Things to Know About Adverse Effects</a>.)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
If the word "effect" is over-used, the word "<a href="http://en.wikipedia.org/wiki/Hypothesis">hypothesis</a>" is under-used. Although generating hypotheses is a critical part of science, hypotheses aren't really marketed as what they are: ideas in need of testing. Often the language is that of attribution throughout, with a little fig-leaf of a sentence tacked on about the need for confirmatory studies. In fact, we cannot take <a href="http://pps.sagepub.com/content/7/6/645.abstract">replication and confirmation</a> for granted at all.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<br />Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com1tag:blogger.com,1999:blog-6353097553819934624.post-63245570542517456622013-06-30T22:44:00.004-04:002022-10-03T20:16:27.556-04:00Goldilocks and the three reviews<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNA3le9gDF4KPivlF3uilLw8EIYO4oTvkd4RqmasTYUfnJ4SqLk7Fb9H2ruiTAOMOatG8M8S4lDJdK9gyOiCIkFFloEjcBsAdGWj4p4n-AhQFd9T7fMfcrsModfZRzzAw_pPyz862WwFhD/s1600/Goldilocks-3-reviews-small.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNA3le9gDF4KPivlF3uilLw8EIYO4oTvkd4RqmasTYUfnJ4SqLk7Fb9H2ruiTAOMOatG8M8S4lDJdK9gyOiCIkFFloEjcBsAdGWj4p4n-AhQFd9T7fMfcrsModfZRzzAw_pPyz862WwFhD/s1600/Goldilocks-3-reviews-small.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Goldilocks is right: that review is FAR too complicated. The methods section alone is 652 pages long! Which wouldn't be too bad, if it weren't that it is a few years out of date. It took so long to do this review and go through rigorous enough quality review, it was already out of date the day it was released. Something that <a href="http://www.ncbi.nlm.nih.gov/pubmed/17638714">happens often enough</a> to be rather disheartening.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
When methodology for systematic reviewing gets overly rococo, the point of diminishing returns will be passed. That's a worry, for a few reasons. For one, it's inefficient and <a href="http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000326">more reviews could be done</a> with the resources. Secondly, more complex methodology can both be daunting, and it can be <a href="http://www.ncbi.nlm.nih.gov/books/NBK98221/">hard for researchers to accomplish with consistency</a>. Thirdly, when a review gets very elaborate, reproducing or updating it isn't going to be easy either.</div>
<br />
It's unavoidable for some reviews to be massive and complex undertakings, though, if they're going to get to the bottom of massive and complex questions. Goldilocks is right about review number 2, as well: that one is WAY too simple. And that's a serious problem, too.<br />
<br />
Reviewing evidence needs to be a <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0005078/">well-conducted research exercise</a>. A great way to find out more about what goes wrong when it's not, is reading <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0050892/">Testing Treatments</a>. And see more on this <a href="http://statistically-funny.blogspot.com/2013/04/look-ma-straight-as.html">here at Statistically Funny</a>, too.<br />
<br />
You need to check the methods section of every review before you take its conclusions seriously - even when it claims to be "<a href="http://statistically-funny.blogspot.com/2012/04/evidence-based-is-new-natural.html">evidence-based</a>" or systematic. People can take far too many shortcuts. Fortunately, it's not often that a review gets as bad as the second one Goldilocks encountered here. The authors of <a href="http://www.ncbi.nlm.nih.gov/pubmed/17376017">that review</a> decided to include only one trial for each drug "in order to keep the tables and figures to a manageable size." Gulp!<br />
<br />
Getting to a good answer also quite simply takes some time and thought. Making real sense of evidence and the complexities of health, illness and disability is often just not suited to a "fast food" approach. As the scientists behind the <a href="http://slow-science.org/">Slow Science Manifesto</a> point out, science needs time for thinking and digesting.<br />
<br />
To cover more ground, people are looking for reasonable ways to cut corners, though. There are many kinds of <a href="http://www.ncbi.nlm.nih.gov/pubmed/18400114">rapid review</a>, including reliance on previous systematic reviews for new reviews. These can be, but <a href="http://www.ncbi.nlm.nih.gov/pubmed/22959594">aren't always, rigorous</a> enough for us to be confident about their conclusions.<br />
<br />
You can see this process at work in the set of reviews discussed at Statistically Funny <a href="http://statistically-funny.blogspot.com/2013/05/he-said-she-said-then-they-said.html">a few cartoons ago</a>. Review number 3 there is in part based on review number 2 - without re-analysis. And then review number 4 is based on review number 3.<br />
<br />
So if one review gets it wrong, other work may be built on weak foundations. <a href="http://www.ncbi.nlm.nih.gov/pubmed/23522971">Li and Dickersin</a> suggest this might be a clue to the perpetuation of incorrect techniques in meta-analyses: reviewers who got it wrong in their review, were citing other reviews that had gotten it wrong, too. (That statistical technique, by the way, has <a href="http://statistically-funny.blogspot.com/2012/10/the-forest-plot-trilogy-gripping.html">its own cartoon</a>.)<br />
<br />
Luckily for Goldilocks, the bears had found a third review. It had sound methodology you can trust. It had been totally transparent from the start - included in <a href="http://www.crd.york.ac.uk/NIHR_PROSPERO/">PROSPERO</a>, the international prospective register for systematic reviews. Goldilocks can get at the fully open review and its data are in the <a href="https://srdrplus.ahrq.gov/">Systematic Review Data Repository</a>, open to others to check and re-use. Ahhh - just right!<br />
<br />
<br />
PS:<br />
<br />
I'm grateful to the Wikipedians who put together the article on <a href="http://en.wikipedia.org/wiki/The_Story_of_the_Three_Bears">Goldilocks and the three bears</a>. That article pointed me to the fascinating discussion of "<a href="http://books.google.com/books?id=tujDvUEpY10C&pg=PA229#v=onepage&q&f=false">the rule of three</a>" and the hold this number has on our imaginations.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<i><br /></i>Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com1tag:blogger.com,1999:blog-6353097553819934624.post-86201964561073239842013-06-23T12:51:00.001-04:002023-08-18T21:53:08.730-04:00Studies of cave paintings have shown....<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKqIs7-OtvqzyqinyN-01m0mzsRB9DB74syUK5VXk56xjYxb2cyIpEOA4YPSEQE8EROi0EnJmscCcRbdKtmK0rS4QHhkJR532BoAa0IzdRwPywIh_SFY0uuDDgk5KTq3LS76I3vp3X-CZQ/s1600/Little-Ogg.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKqIs7-OtvqzyqinyN-01m0mzsRB9DB74syUK5VXk56xjYxb2cyIpEOA4YPSEQE8EROi0EnJmscCcRbdKtmK0rS4QHhkJR532BoAa0IzdRwPywIh_SFY0uuDDgk5KTq3LS76I3vp3X-CZQ/s1600/Little-Ogg.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The mammoth has a good point. Ogg's father is making a classic error of logic. Not having found proof that something really happens, is not the same as having definitive proof that this thing cannot possibly happen.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Ogg's family doesn't have the benefit of <a href="http://plato.stanford.edu/entries/aristotle-logic/#SubLogSyl">Aristotle's explanation of deductive reasoning</a>. But even two thousand years after Aristotle got started, we still often fall into this trap.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
In evidence-based medicine, a part of this problem is touched on by the saying, "<a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2550545/">absence of evidence is not evidence of absence</a>." A study says "there's no evidence" of a positive effect, and people jump to the conclusion - "it doesn't work." Baby Ogg gets thrown out with the bathwater.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The same thing is happening when there are no statistically significant serious adverse effects reported, and people infer from that, "it's safe." </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
This situation is the opposite of the problem of reading too much into a finding of statistical significance (<a href="http://blogs.plos.org/absolutely-maybe/2016/04/25/5-tips-for-avoiding-p-value-potholes/">explained here</a>). Only in this case, people are <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2550545/">over-interpreting non-significance</a>. Maybe the researchers simply didn't study enough of the right people, or they weren't looking at the outcomes that later turn out to be critical.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Researchers themselves can over-interpret negative results. Or they might phrase their conclusions carelessly. Even if they avoid the language pitfalls here, journalists could miss the nuance (or think the researchers are just being wishy-washy) and spread the wrong message. And even if everyone else phrased it carefully, the reader might jump to that conclusion anyway.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
When researchers say "there is no evidence that...", they generally mean they didn't find any, or enough of, a particular type of evidence that they would find convincing. Obviously, no one can ever be sure they have even seen <u>all</u> the evidence. And it doesn't mean everyone would agree with their conclusion, either. To be reasonably sure of a negative, you might need quite a lot of evidence.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
On the other hand, the probability of something being extremely unlikely to be real based on quite a lot of knowledge - that there's a community of <a href="http://plato.stanford.edu/entries/learning-formal/">giant blue swans</a> with orange and pink polka dots on the Nile, say - <a href="http://ojs.uwindsor.ca/ojs/leddy/index.php/informal_logic/article/view/2967/2516">increases the confidence you might have</a> in even a small study exploring that hypothesis.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
In 2020 during the Covid-19 pandemic, we found out how deeply another problem goes: taking the absence of particular types of evidence as the rationale for not taking public health action. Early in April I <a href="https://www.wired.com/story/the-face-mask-debate-reveals-a-scientific-double-standard/" target="_blank">wrote in WIRED</a> about how this was leading us to policies that didn't make sense – especially in not recommending personal masks to help reduce community transmission. At the same time, Trisha Greenhalgh and colleagues <a href="https://www.bmj.com/node/1024432.full" target="_blank">pointed out</a> that was ignoring the precautionary principle: it's important to avoid doing harm caused by not taking other forms of evidence seriously enough. When it was finally acknowledged that the policy had to change, <a href="https://www.npr.org/sections/health-shots/2020/07/01/886299190/it-does-not-have-to-be-100-000-cases-a-day-fauci-urges-u-s-to-follow-guidelines?utm_medium=RSS&utm_campaign=news" target="_blank">it was a recipe for chaos</a>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Which brings us to the other side of this coin. Proving that something doesn't exist to the satisfaction of people who perhaps need to believe it most earnestly, can be quite impossible. People trying to disprove the claim that vaccination causes autism, for example, are finding that despite <a href="http://plato.stanford.edu/entries/enlightenment/">the Enlightenment</a>, our rational side can be vulnerable to highjacking. <a href="http://plato.stanford.edu/entries/voltaire/">Voltaire</a> hit that nail on the head <a href="http://goo.gl/GGGvK">in the 18th century</a>: "The interest I have to believe a thing is no proof that such a thing exists."</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhuyGvCX4-YBHt6XGgKR5F52pCTfXZBaKFLtaSXFNVmT71lCeLUkdkP5xbqQj-uajcaep14Xe-Ay9QNC8eeA6DhQ7VqLPcYUsJ2oE4WgPlmIK80aD8AWXz4kR0MnWEfXtJ9GZF1cv7Q3kxOB4QTIAv3hCtfFlDwQJVmCMPI4o22f_poXxq-nKFFzZ57TG1/s1976/Voltaire.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Voltaire quote from 1763 with a cartoon pic of a man from that period: "The interest I have to believe a thing is no proof that such a thing exists."" border="0" data-original-height="603" data-original-width="1976" height="98" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhuyGvCX4-YBHt6XGgKR5F52pCTfXZBaKFLtaSXFNVmT71lCeLUkdkP5xbqQj-uajcaep14Xe-Ay9QNC8eeA6DhQ7VqLPcYUsJ2oE4WgPlmIK80aD8AWXz4kR0MnWEfXtJ9GZF1cv7Q3kxOB4QTIAv3hCtfFlDwQJVmCMPI4o22f_poXxq-nKFFzZ57TG1/w320-h98/Voltaire.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i>~~~~</i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i>Update 3 July 2020: Covid-19 paragraph added.</i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
<div class="separator" style="clear: both; text-align: left;">
<i><br /></i></div>
Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com2tag:blogger.com,1999:blog-6353097553819934624.post-51052743662457213782013-05-21T17:13:00.002-04:002022-10-03T20:17:05.422-04:00He said, she said, then they said...<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLo4DPlEeJu65u5UF2d_FDV7EUvyXcRB11tYi-hsdIoXj4qItY48qa7UBI-1B6KaN95ywK6bFVCk1YeGitBnRNRwxNa991vzBfAwhOu_Www3hHThLp6fi-Qo3HiyX8bqgAL70XTdiNNrLH/s1600/He-said-she-said.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="361" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLo4DPlEeJu65u5UF2d_FDV7EUvyXcRB11tYi-hsdIoXj4qItY48qa7UBI-1B6KaN95ywK6bFVCk1YeGitBnRNRwxNa991vzBfAwhOu_Www3hHThLp6fi-Qo3HiyX8bqgAL70XTdiNNrLH/s1600/He-said-she-said.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Conflicting studies can make life tough. A <a href="http://statistically-funny.blogspot.com/2013/04/look-ma-straight-as.html">good systematic review</a> could sort it out. It might be possible for the <a href="http://statistically-funny.blogspot.com/2012/10/a-dip-in-data-pool.html">studies to be pooled</a> into a <a href="http://statistically-funny.blogspot.com/2012/10/the-forest-plot-trilogy-gripping.html">meta-analysis</a>. That can show you the spread of individual study results and what they add up to, at the same time.<br />
<br />
But what about when systematic reviews disagree? When the "he said, she said" of conflicting studies goes meta, it can be even more confusing. New layers of disagreement get piled onto the layers from the original research. Yikes! This post is going to be tough-going...<br />
<br />
A group of us defined this <a href="http://www.imbi.uni-freiburg.de/OJS/cca/index.php?journal=cca&page=article&op=view&path%5B%5D=6977">discordance among reviews</a> as: the review authors disagree about whether or not there is an effect, or the direction of effect differs between reviews. A difference in direction of effect can mean one review gives a "thumbs up" and another a "thumbs down."<br />
<br />
Some people are surprised that this happens. But it's inevitable. Sometimes you need to read several systematic reviews to get your head around a body of evidence. Different groups of people approach even the same question in different but equally legitimate ways. And there are lots of different judgment calls people can make along the way. Those decisions can change the results the systematic review will get.<br />
<br />
When and how they searched for studies - and what type and subject - means that it's not at all unusual for groups of reviewers to be looking at different sets of studies for much the same question.<br />
<br />
After all that, different groups of people can interpret evidence differently. They often <a href="http://www.ncbi.nlm.nih.gov/books/NBK98221/">make different judgments about the quality of a study </a>or part of one - and that could dramatically affect its value and meaning to them.<br />
<br />
It's a little like watching a game of football where there are several teams on the field at once. Some of the players are on all the teams, but some are playing for only one or two. Each team has goal posts in slightly different places - and each team isn't necessarily playing by the same rules. And there's no umpire.<br />
<br />
<div class="MsoNormal">
Here's an example of how you can end up with controversy and people taking different positions even when there's a systematic review. The area of some disagreement in this subset of reviews is about psychological intervention after trauma to prevent post-traumatic stress disorder (PTSD) or other problems:<br />
<br />
<a href="http://www.ncbi.nlm.nih.gov/pubmed/12143079">Published in 2002</a>; <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0010736/">Published in 2005</a>; <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0015848/">Published in 2005</a>; <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0014194/">Published in 2010</a>; <a href="http://www.who.int/mental_health/mhgap/evidence/resource/other_complaints_q5.pdf">Published in 2012</a>; <a href="http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0055866/">Published in 2013</a>.<br />
<br />
The conclusions range from saying debriefing has a large benefit to saying there is no evidence of benefit and it seems to cause some PTSD. Most of the others, but not all, fall somewhere in between, leaning to "we can't really be sure". Most are based only on randomized trials, but one has none, and one has a mixture of study types.<br />
<br />
The authors are sometimes big independent national or international agencies. A couple of others include authors of the studies they are reviewing. The definition of trauma isn't the same - they may or may not include childbirth, for example. The interventions aren't the same.<br />
<br />
The quality of evidence is very low. And the biggest discordance - whether or not there is evidence of harm - hinges mostly on how much weight you put on <a href="http://www.ncbi.nlm.nih.gov/pubmed/9328501">one trial</a>.<br />
<br />
It's about debriefing. The debriefing group is much bigger than the control group because they stopped the trial early, and <a href="http://www.ncbi.nlm.nih.gov/pubmed/16684642">while it's complicated</a>, that can be a <a href="http://www.ncbi.nlm.nih.gov/pubmed/20332404">source of bias</a>.<br />
<br />
The people in the debriefing group were at quite a lot higher risk of PTSD in the first place. Data for <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1444839/">more than 20%</a> of the people randomized is missing - and <a href="https://www.iqwig.de/download/Vortrag_High_dropout_rates_in_trials_included_in_Cochrane_Reviews_.pdf">that biases the results</a> too (it's called attrition bias). You can't be sure those people didn't return because they were depressed, for example. If so, that could change the results.<br />
<br />
It's no wonder there's still a controversy here.<br />
<br />
<br />
<i>See also my <a href="http://blogs.plos.org/absolutely-maybe/2017/07/03/5-tips-for-understanding-data-in-meta-analyses/">5 tips for understanding data in meta-analysis</a>.</i><br />
<i><br />
Links to key papers about this in my comment at PubMed Commons (archived <a href="https://via.hypothes.is/https://europepmc.org/abstract/MED/12076399#annotations:a-MKYn42EeiEUYcfdjxzXw" target="_blank">here</a>).</i><br />
<i><br />
If you want to read more about debriefing, here's my post in Scientific American: <a href="http://blogs.scientificamerican.com/guest-blog/2013/05/21/dissecting-the-controversy-about-early-psychological-response-to-disasters-and-trauma/">Dissecting the controversy about early psychological response to disasters and trauma</a>.</i><br />
<i><br /></i></div>
<!--EndFragment-->
<!--EndFragment-->Hilda Bastianhttp://www.blogger.com/profile/01418954331826160477noreply@blogger.com0