Lies, Damn Lies, and Meta-Analysis

I’ve always been interested in how statistics are used in social sciences and the popular media. It was Mark Twain who famously said “there’s lies, damn lies and statistics,” but of course statistics never lie, only people. Business Week recently reported on an arcane branch of statistical analysis, meta-analysis, that is being used more and more to justify claims about new drugs, herbal remedies, educational systems and just about anything else that is a popular field of study and can easily be reduced to a set of numbers. It’s definitely contributing to the ‘statistics lie’ debate.

I’m no statistician, but here’s my understanding of meta-analysis. There are many fields of study where research is repeated, either to attempt to verify or replicate existing studies, contradict findings, or get a richer or slightly different understanding of a question. Medical research and educational research are good examples – there’s been numerous experiments done to test the efficacy of vitamin C as a cold cure, for example. Often individual studies aren’t conclusive, or seem to disagree: but what if you could pool the research results of multiple studies and get an aggregated, summary set of findings? This would be very cool, especially if you’re an impoverished PhD (are there any other kind?) with no budget but access to a well-stocked library: all you do is judicially sum reported results from a handful of published papers, with no fieldwork required. Even better, the results of meta-analysis should increase the power of the overall conclusions – you can discern smaller effects, based on bigger sample sizes.

All well and good, but two problems emerge. First, very few research studies are conducted identically. Questions are asked in a slightly different way, to a slightly different population, under slightly different circumstances. Sometimes these differences can be reasonably ignored because they are small or insignificant, but most often this is a judgment call. Make the wrong call, and you are comparing apples and oranges. A second problem is what is often called the file drawer effect: research that is inconclusive isn’t likely to be published, and so goes unreported, potentially biasing meta-analysis. This can result in exaggerated effects being reported.

This helps explain why we see heated debates about the impact of standardized school tests, allergy remedies, or the impact of violent TV on real-world violence.

Sometimes I wonder if blogging is simply meta-comment, or meta-news, or meta-babble. In the same way that meta-analysis is data from data, or more accurately statistics calculated from other statistics, it feels like blogging is derived, secondary, imitative. It has all the power of meta-analysis, and all the pitfalls.