For any particular health behavior or condition, the number of research studies is ever-growing. The expansive literature makes it nearly impossible for health practitioners, and even researchers, to stay up-to-date.
Meta-analyses are a type of systematic review that allow for the combination of findings from individual studies in a way that increases statistical power and may thus generate evidence-based ‘bottom lines’ for practice. However, a recent viewpoint in the Journal of the American Medical Association, The Misuse of Meta-Analysis in Nutrition Research, leaves us wondering whether meta-analyses do more harm than good.
Some of the most common flaws discussed in this viewpoint include:
- The people. Individual studies may include a range in demographic characteristics, like age, sex, race, and ethnicity. While it is typically a good thing to include a variety of people in a single study, trying to compare different study populations can make it more challenging to identify real effects. Think: comparing a study that looked at egg consumption and cholesterol levels in men aged 65 and older to a study looking at women aged 20 – 40 years – there are many other factors that could explain the observed effects.
- The study design. Although studies may be looking at the relationship between saturated fat and heart disease, they may have used different tools to measure saturated fat intake over varying periods or time or different measures related to heart disease. In addition, some trials may have randomly assigned participants to a group while others followed their natural behaviors over time. This is like trying to compare apples and oranges, although they are both fruit, they are in fact different and it may not be appropriate to try and interpret them together.
Results of meta-analyses matter because they can influence health care policy – either by providing an evidence base for decision-making and/or media headlines prompting public conversation that elevates the priority of a specific condition or behavior. Barnard and colleagues suggest the peer-review process should and could be improved by:
- Having content expert editors as well as editors with expertise in meta-analysis techniques
- Having authors of the review confirm the appropriateness of the representation of the data with authors of the original report
- Having transparent methods and data so that others may reproduce the analysis
- Pooling original primary data and not published summary data
Meta-Analysis. Study Design 101. https://himmelfarb.gwu.edu/tutorials/studydesign101/metaanalyses.html
Barnard ND, Willett WC, Ding EL. The Misuse of Meta-analysis in Nutrition Research. JAMA. Published online September 18, 2017. doi:10.1001/jama.2017.12083
The Nutrition Source, Harvard T.H. Chan School of Public Health. Meta-analyses in nutrition research: sources of insight or confusion?