Systematic Reviews and Meta-Analysis

it’s a discussion of meta-analyses and we’re going to talk about not only what they are and how to evaluate them but we’re going to discuss how to do them and then right after the break and in an hour or two we’re gonna actually do the meta analyses and you’ll I think be persuaded that there’s no real magic in these in the sense that they’re easy to do you have the tools in your laptop now how to do them okay so we’re going to review these a little bit so here’s what I’m planning to talk about and you stop me at any point if I’m not covering stuff or not being clear but we’re going to talk about these systematic reviews and meta-analyses that’s SR and MA and we’re going to talk both about how to appraise other people’s systematic reviews and how to do them ourselves and we’ll look at some threats to their reliability the same way we have four single studies okay and we’re going to go about it the same way that we we just did for single randomized control trials always keeping in mind this hierarchy of evidence and a lot of people put this meta-analysis at the top okay I’m not sure that that’s uniformly agreed to but but there’s certainly powerful ways to answer questions always important and we’ve already said this this morning and before this is not the only feature we use to make decisions and recommend therapy okay our own experience isn’t thrown out the window okay and neither is patient judgment or values or those things really important to know that the the practitioner of evidence-based medicine advocates integrating all of those things it’s not evidence only medicine so with that in mind systematic reviews and meta-analyses are almost the same they’re concise summaries of our best evidence and they use this very rigorous method that’s explicit and transparent so that you can reproduce what what other authors have done to to collect appraised and then synthesize a result from the evidence about a specific question in a way that minimizes both bias and random air and we’ll talk about how how that’s done but if they’re done right they’re supposed to look at the whole truth all of the evidence that we have about a particular question and the only difference between a systematic review and meta-analysis is that the meta-analyses introduced some quantitative techniques to help us and that’s what we’ll do when we do the hands-on stuff okay so you you you fill this table out for me so with regard to the question how does it how does a systematic review differ from this sort of narrative review which which I’m using as a example of the book chapter how are they different with regard to the question yeah exactly heiko versus focused is how they differ with review fearing so for example you might have a question about does prophylactic anticonvulsant medication prevent first seizures in patients with subarachnoid hemorrhages or craniotomy that’s a very good focused question for for a systematic review you might see a chapter of anticonvulsant management in the neurosurgery patient so a broad question a focus question how about sources and search for the evidence differences and similarities so a narrative review they don’t generally tell you they select the articles there these chapters with 200 references but you don’t quite know what the selection or the filter or the criteria were but it’s very comprehensive and explicit in a systematic review selection of articles basically the same answer right very explicit in a systematic review and potentially biased in a chapter because you have an idea about what the answer to a question is and you can select the evidence to support that it’s like writing you know high school English papers you know the that five paragraph kind of paper that you how about the appraisal of the evidence in a systematic reviewer in a chapter let’s say yeah just the way that we just did exactly and the synthesis is quantitative if you can and qualitative with very within very rigid guidelines in a systematic review and as a result the inferences are occasionally evidence based in a chapter but always evidence based in a systematic review so the whole process is really similar when we’re looking at single studies in

meta-analyses and and I’m going to suggest that we think about meta analyses and one of the groups actually has selected a meta-analysis for their project think about a meta-analysis as a clinical trial where your patients are the individual articles or the evidence that you’re going to collect for the meta-analysis and that that helps clarify with what we’re doing and you can do these meta-analyses for therapy and for diagnosis and prognosis and and you know risk factors that their applicability to any type of question okay so just to just to kind of review these systematic reviews compared to a narrative like a chapter limit by us they’re transparent rigorous and reproducible if you do a meta-analysis and I do one on the same question we ought to get the same answer so you can check my work their comprehensive which of these is the most important thing the thing that meta-analyses really do the best the reason in fact that we we do them at all I mean all of these are good things but what what what thing does meta-analysis give to you that just looking at the individual articles don’t it is comprehensive but I could look at the articles without meta analyzing them I could be comprehensive yeah it increases the precision that’s what meta-analyses do they don’t make the studies any better right if they’re bad studies and you do a meta-analysis there’s still bad studies they don’t increase your estimate of effect size they narrow your confidence interval they increase the precision those are all synonyms and like clinical trials right I asked you to think about meta analyses as clinical trials they’re at risk for ran our bias and confounding at every step right so we’ll talk about that but systematic reviews and meta-analyses have some particular risks of bias that aren’t relevant to single studies and those in particular our publication bias and heterogeneity and we’ll talk about those and like single trials where we use consort there are a whole bunch of published easy to access ways to both evaluate the meta analyses and to produce your own both the strategy for getting the data and the quantitative aspects prisma is an example there a whole bunch and we’ll talk about those okay but meta-analyses only do that and primarily they increase the precision of your effect size estimate how much does early anticonvulsant therapy in patients with first seizures reduce the risk of seizures from broad to narrow that it increases the precision doesn’t improve the quality of the evidence that goes into them okay but if you if you if you do these properly then you then you have a document that provides you the best evidence and if you do them quantitatively you actually get a number a number of Association odds ratio a relative risk you can certainly translate those into number needed to treat that we can make use of in the clinic so how do you actually do these well the easiest way is to find someone else who’s done them and you can find these in big databases of meta-analyses maybe the easiest way but medline also provides meta analyses and if you’re using Ovid there’s a there’s actually a meta-analysis button you can just that it makes it really easy it’s almost as easy if you’re using PubMed this clinical queries page will get you right to meta analyses and so you can you can access those with two clicks and instead of one and remember try and look at other databases beside pubmed if you’re if you’re doing something formal like a practice guideline for the double ANS okay so how do we actually do these and this is the like the instructions for the small group session so we’ll get started the way we get started with any of these projects by asking a really focused question using the Pico format and remember we’re thinking about meta analyses as clinical trials with our patients as the articles that we’re putting in okay so you have to have pre-established inclusion and exclusion criteria the same way you would say patients greater than 18 years of age with a you know with with I don’t know with a performance status of whatever you’ve got to do that with the articles why why not just see what you get and then pick the articles why why do you insist on having established selection criteria i means the same reason you do that in a clinical trial yeah exactly bias is a risk at every step and this is

a huge potential for bias studies when you’re trying to combine them aren’t necessarily going to be different so this is the second big risk in these meta-analyses right so the patient groups are always a little bit different the age may be different the criteria for performance the type of disorder that the interventions may vary a little bit the outcome measures frequently are not exactly the same and so the studies are always heterogeneous and so you also have to anticipate that they’re not going to be the same and explain how you’re going to deal with that how much heterogeneity are you going to permit before you say I can’t combine these studies okay you need to know all of that stuff ahead of time and then you do your exhaustive literature search and these are the components people advocate so not just medline but other databases talk to experts look at a list of randomized control trials to see whether you know clinical trials gov to see whether maybe you’re missing some randomized trials that have been done many people will accept published abstracts that’s a little controversial but you’ve got to look through the list of references in your selected articles have to be really exhaustive about this and then apply those inclusion and exclusion criteria that that you’ve pre-established to select these these um final eligible articles and then critically appraise them like what we just finished doing why is it important to make the selection and the critical appraisal by the way in duplicate exactly and and how are you if if you and I both appraise the articles how are we going to decide how frequently we agree what’s the statistical way to do that yeah and you can by the way even though our tool looks at capital with two evaluators you can you can apply kappa two three or four so it’s doable yeah so you want to reassure us that that nick and i are seeing the same thing that our Silla and you should have a strategy when we disagree how are you going to resolve the disagreement beforehand we’re going to discuss it and resolve our differences we’re going to get a third person to break the tie you shouldn’t you should have that all established ahead of time and you can do these for therapeutic questions like we’ve been discussing or for diagnosis or prognosis meta-analyses apply to any type of question okay so it’s just reviewing what I just said about two people doing all of these steps because like any clinical trial meta analyses are subject to random error and bias and we want to minimize those at every step and then if you can we want to want to quantitatively evaluate the evidence and we’re going to now talk about what quantitation means okay we’re going to talk about the math so here’s what a meta-analysis looks like this this is data from an from the early aids trials to antibiotics and and these are the two ways that you see meta-analysis displayed this is all of the individual studies and the meta-analysis and this is a technique called accumulative meta-analysis where your there’s the first study and you’re adding the second one here’s the total number and you’re adding another one the numbers gone up um what observation do you see about the maybe it’s the easiest to see here but but also there about what will tell me what this is what what is this picture describing what’s the diamond and what’s the arms on either side yeah exactly this is your point estimate and your confidence interval do you see a consistent association between the size of the confidence interval and the size of the study yeah are you surprised about that and that’s why the the smallest is is the summary statistic at the bottom and that’s why the cumulative meta-analysis looks like it does bigger the size of the study the more precise the estimate what did I say wrong so I know from from sitting over there that that you can’t always hear what dr. Barker has said so in this particular example a big reason that that the summary statistic is not as small as that as the Concord study is the heterogeneity of effect estimates and precision of the trials that go into the meta-analysis and it also has to do with the technique that people use to do the meta-analysis and a few other things but by and large big studies are the way that you get confidence interval smaller increased precision another example that really shows the same thing okay this is a streptokinase for acute MI so we perform the meta-analysis as we’ve discussed to try and get a better assessment of the true effect size and

the studies that we put in variant precision and we sort of have to wait them so that we but soaps a why for example wouldn’t you just I mean we we have we have all these numbers if you bothered to look up the articles you could see how many patients met and didn’t meet an endpoint and you could just stick them all add them up stick them into your two by two table why wouldn’t you do that much simpler the math is certainly much simpler yeah yeah so a patient a patient from a tiny study shouldn’t probably be weighted the same as a patient from a monster study okay you need some way to to credit studies for for their size because bigger studies generally have a better estimate of effect size and so that’s what meta-analyses do and they’re they’re a bunch of strategies for doing those the two common ones are the top two and those are the only two I think we’ll talk about today if you’re interested we can discuss some of the other ones but these are the ones that you’ll see when you read papers and that you’ll use probably this afternoon when you’re doing your own so here we’re going to come back to this but just for a second look at the this is a what’s called a fixed effects model of doing that analyses it’s the one that that will provide you in the tools this is called a random effects model when you get down to the bottom you get pretty much the same result look look for a minute and you’re going to see the same side again look for a minute and compare side to side and get in your mind what what do you see some some difference is there between the two strategies the data is the same any any differences that stand out to you yeah yeah that’s really subtle and good observation so this is a more conservative estimate of the effect size there’s something that goes along with that to another subtle observation yeah the confidence intervals of the random effects cross the midline more often and what does that you know what think about that one I’m going to give you three three or four slides to distill that that’s that’s good and while we’re doing that I’ll talk about the fixed and random effects model so they’re there to sort of ways to think about how you would put studies together okay a fixed effect model makes the assumption that there’s one true effect size for the specific question in the universe and each of the studies that you’re putting together in your meta-analysis kind of estimate that one true effect size and if you could do an infinitely large study that included all of the patients with a who have data relevant to your question you would get precisely that effect size estimate but because you can’t because we have to sample the studies are all different they’re there they don’t give you the same precision and effect size and so a fixed effects model will wait the different estimates based in some way on sample size inverse variance some some way to wait credit sample size in the different studies and so what what fixed effects model is recognizing is this within study random error is that fair we’re sampling from a pool of possible groups of patients to answer a question each of the studies is a different sample they’re going to have some random variation of the effect size and so we’ve got a got to take that into account in our meta-analysis and it doesn’t really address this issue of heterogeneity of between study variation that’s the fixed effects model the random effects model is a little bit different the assumption here is that each study in your in your random effects model is estimating a different but true effect size because each of the studies has slightly different populations and slightly different interventions and slightly different outcome measures and so each study is estimating something a little bit different but you still want to put them together we still need a you know we still need direction and taking care of our patients and and so when you combine those you now have the source of error thats related to randomness because you’re doing a selection of all the possible patients and this between study variability because each study is estimating asking a slightly different question estimating a slightly different or maybe a very

different effect size and so the the strategy that that the random effects model uses accounts for both of those and and what it really does is it it does the fixed effects model this sort of waiting based on the larger or smaller study and then it goes back and looks at how much of this of this difference between studies is there and it undo undoes to some extent the the waiting that you’ve just done based on size so that if there’s a lot of variability between studies you undo it completely and it’s basically like just sticking numbers into the two by two table and if there’s if there’s really almost no variability then you don’t undo it at all and those are the two ranges it’s a fixed effects model random error and as a result the as you observed the random effects model kind of weights smaller studies more highly and it’s more susceptible to publication bias which we’ll we’ll take a look at but less susceptible to heterogeneity and as a result produces these wider or more more conservative confidence intervals so that’s exactly what you observe your some pictures just to see same question okay by phosphonates for hip fractures to prevent hip fractures so here’s the fixed-effects calculation the random effects okay really gets sort of the same answer in this particular case but with a more conservative estimate of effect size but but they can the models can sometimes actually differ in terms of their instruction to you so random effects fix the facts very different answers so back to the one that we showed you examine alee sort of following from the more conservative and less conservative estimates if you were using this model you would have stopped your and you were following this along cumulatively you could have stopped your investigation a couple of years earlier than you would have if you were using the random effects model because it’s a more conservative estimate anyway that’s just a kind of a summary of the two side by side as I say we we use for our own tool fixed effects model people will sometimes use the random effects model usually there’s not a big difference but you should be aware of the circumstances where where it is important oh you know the other cool thing about uh about that study is this this is actually a way of we can’t usually we said we can’t measure bias precisely this is at least giving you the maximum quantity of heterogeneity you can actually measure in that sense how big could it possibly be the difference between those two I think that’s kind of cool and then there’s some unique biases or threats to reliability with these two with meta now a meta-analytic techniques one of them is publication bias and you probably know this really familiar to you what what is publication bias in general so positive outcome studies more likely to be published what other studies either are more likely or less likely English language studies were more likely to get published so small- non-english language studies tend either not to get published or if they do get published there’s a big delay a bigger delay in their publication okay and there are a bunch of strategies to to assess that in meta-analyses the one that’s that’s kind of the coolest in a way they all have cool names though but the one that’s the coolest and the one that you see a lot is this thing called a funnel plot and it’s probably worth spending a minute deciding what this is all about so what you do basically this is the summary estimate of the effect size that that down pointing arrow and then we’ve just plotted the different studies the risk ratio on the x-axis and some measure in this case the actual sample size and then you try and fit a triangle a symmetric triangle that includes most of the studies and is centered at this estimate what is the try what is why is it a triangle why isn’t it a some other shape what properties of what you’re doing there make this a triangle why isn’t it a funnel why do they call it a funnel plot but why is it why is it why is it that shape yeah exactly the larger

the sample size the less exaggerated the effect size is and the smaller the study further out so this is a neat way of identifying outliers for one thing and it also is a really good visual way of identifying the potential for missing study so remind me again what type of studies you think are not are potentially not going to be included in your in your search for articles or haven’t been published and where would that be on the on the picture here yeah where that hole is that’s what a funnel plot does for you this funnel plot would make you concerned that there may be some publication bias that you’re missing some small negative studies just other examples okay we actually have one of these that we’ll look at in journal club um so anyway there are other strategies and they have such cool names that I figured it was worth just doing two slides this this failsafe strategy asks the question how many undetected negative studies would you have to add into your meta-analysis make up and add in to change the result okay I knew you would we would ideally like that to be not a small number if a small number of those trials change your result then you’re concerned about the possibility of publication bias and similarly this file drawer test you sequentially remove studies from your meta-analysis and you’d like to find that no one or two studies have a huge impact on the outcome of your meta-analysis if they do again you’re worried that that your estimate is tenuous and then the other one heterogeneity we’ve talked enough about this so that you remember that studies differ in all sorts of features and if they’re different enough you worry about whether it’s honest fair and it permissible to combine them so let’s think about some some combination rules because the studies are you know they they’re inevitably going to be different okay but we still want the best estimate of an effect size so what do you think about this rule you’re comfortable pooling if the studies are on the same side of do you like that rule so would you pull these studies yeah comfortable with these how about these same distance apart or why so so pulling these or no I probably be worried about this right they look like they’re estimating they’re grouped and they look like the two studies and the three studies are estimating some different effect perhaps asking some different question different patient population different outcome measure different intervention different enough that they’re giving you a very different estimate of effect size how about these pretty comfortable with ease right although one might say that a meta-analysis was superfluous in this really tightly grouped set of studies how about these same distance right the same same thing as that probably this is where a meta-analysis would be most useful it looks like they’re all estimating the same effects visually anyway but it’s so imprecisely don’t know so this this is a good perhaps but I think you can’t focus exclusively on on where we’re in the graph of benefit or harm how about taking into consideration the confidence intervals so we were a little bit divided about this one how about now so now now jeez these confidence intervals completely overlap so apart from what you might say about the studies themselves yeah but this is this is something that a meta-analysis would reduce those confidence intervals give you more precision on your estimate and you can’t really make a good case that they’re estimating something different or estimating something at all so anyway there there are other tests for heterogeneity this one the Cochrane Collaboration has now adopted this and so it’s become a little popular but this this idea of a test for heterogeneity that uses a p-value so the the null hypothesis here is that the underlying treatment effect is the same in all of your studies and you do this statistical test and so a low p value means what that’s your null hypothesis you get a low p-value is that what you want to see or not I guess it depends what your interest is but a low p-value suggests the chance is not the best explanation for the variability that you’re seeing and so you may not be able to or want to combine those um this um almost all of

the math behind meta-analyses is not as secure as the math behind two by two tables and now analyzing individual studies there’s a lot of uncertainty I don’t want to say fudging and so for example it’s reflected here by calling a important p value something that we would never consider important in a single study and in general the math is less well defined especially as you get out into some of these other things heterogeneity so so it’s a problem you should recognize it and you should have a strategy for dealing with it but it’s not always avoidable and hear that here are the options you can ignore it if it’s not substantial and just go ahead and analyze you can account for it with a different model of meta-analysis you can say gee it’s not possible to do a meta-analysis yet with the data that’s available and then there’s some more sophisticated kind of analogous to multivariate analysis strategies that you can use and these are the ones that are that are still not well established okay so on the one hand heterogeneity is always there it’s a threat to the reliability of your meta-analysis you need to recognize and account for it but we still need to have information that makes use of all the data that’s available there are lots of ways to address that so pretty much finished so a good meta-analysis does a lot of good things for you most importantly it gives you a more precise estimate of the effect size and because the numbers are big it allows this sort of hypothesis generating approach to subgroup analysis that you can’t frequently do with single studies but it doesn’t do everything that doesn’t supersede good clinical judgment and it doesn’t make studies better if they’re not good okay so bad studies don’t get improved by doing a meta-analysis bad studies are still bad studies you