Study identification and selection

We searched Medline, Embase, Cochrane Database of Systematic Reviews, and Cochrane Central Register of Controlled Trials between 1 January 1985 and 10 March 2013 using the search terms atorvastatin, fluvastatin, simvastatin, lovastatin, pravastatin, rosuvastatin, cholesterol, cardiovascular disease, and hydroxymethylglutaryl-CoA reductase inhibitors/therapeutic use. We supplemented electronic database searches by manually cross checking reference lists from published trials and review articles. Two researchers independently performed abstract, title, and full text screening. A third researcher approved study selection.

As described previously,14 15 16 17 we included open label and double blind randomised controlled trials comparing one statin with another at any dose or with control (placebo, diet, or usual care) for adults with, or at risk of developing, cardiovascular disease. We included both fixed dose and titration trials of atorvastatin, fluvastatin, lovastatin, pravastatin, rosuvastatin, and simvastatin if they had more than 50 participants per trial arm, lasted longer than four weeks, and reported changes from baseline low density lipoprotein (LDL) cholesterol concentration. In accordance with predefined criteria,24 we excluded trials conducted in patients with renal insufficiency.

The primary outcome of interest was the difference in mean change from baseline in serum LDL cholesterol concentration between two comparator treatments for a given dose (change from baseline in the treatment group minus that in the control group). We extracted information on trial and patient population characteristics, and the outcome of interest. We extracted the total number of randomly assigned participants unless only a subsample of patients were eligible for statin therapy; this was the case in some of the large trials, which measured LDL cholesterol levels only in a subsample of the original trial population. One researcher extracted data and another independently checked for accuracy.

Categorisation of potential bias

We explored two sources of bias. First was quality related bias, which we defined as the potential underestimation or overestimation of the true intervention effect due to flaws in the methodological attributes of randomised controlled trials.3 Second, we explored industry sponsorship bias, which could occur when a pharmaceutical sponsor favoured its own drug in placebo controlled or active comparator trials or the highest dose of its own drug in dose comparator trials.23

To evaluate quality related bias, we extracted data on the methodological quality of included randomised trials at the study level.3 Our assessment of methodological quality included published trial protocols when such documents were cited in the original trial publications. Using the risk of bias tool developed by the Cochrane Collaboration,25 we collected information on blinding (that is, did the investigators blind trial participants and researchers from knowledge of which treatment a trial participant received?); random sequence generation (that is, were the methods for allocation sequence reported to determine whether it produced comparable groups?); allocation concealment (that is, were the methods used to conceal the allocation sequence reported to determine whether group allocations could have been foreseen before or during randomisation?); blinding of outcome assessment (that is, did trial investigators blind outcome assessment from knowledge of which intervention a participant received?); indications of incomplete outcome data (that is, did the investigators report completeness of outcome data for LDL cholesterol lowering, including attrition and exclusions from the analysis?); and indications of selective reporting (that is, did the investigators fail to report tolerability and harm outcomes commonly reported in randomised trials of statins (such as withdrawals due to adverse events, creatine kinase elevations, hepatic transaminase elevations, or myalgia), and were there deviations in trial outcomes from published protocols?). On each methodological attribute, we gave studies a rating of high, low, or uncertain quality. For each trial, we assigned one point for each item with “high quality” to calculate the overall methodological quality score, ranging from 0 (worst methodological quality) to 6 (best methodological quality). Using this information, we evaluated the distribution of methodological quality of different comparisons across the evidence network (for example, explored whether trials of simvastatin v control had on average higher or lower methodological quality than those of atorvastatin v rosuvastatin).

We then extracted information on the funding sources of included trials. Two potential funding sources were industry and non-industry (such as government, academic institutions, other not for profit research organisations). Industry sources included any private, for profit, pharmaceutical company involved in research and development, manufacturing, or marketing of statins. Most of the included trial publications contained a statement delineating the funding sources; in cases where sponsorship information was not clearly documented, we checked the author affiliations and categorised the studies with industry affiliated authors as industry sponsored. Trials with industry, government, and/or academic institution co-sponsorship were categorised as industry sponsored (unless the trial investigators included a statement that the funding body had no involvement in trial design, conduct, analysis, or reporting).

To explore the presence of design bias, we compared the percentage of industry sponsored trials with placebo controls with the corresponding percentage in non-industry sponsored trials. In addition, we compared the type of drug and dose comparisons between industry and non-industry sponsored trials. We considered industry sponsored trials where investigators included a statement that the funding body had no involvement in trial design, conduct, analysis, or reporting as industry sponsored for the evaluation of design bias.

Finally, we identified which statin (at a specific dose) was favoured in a given trial. For the exploration of industry sponsorship bias, we had two a priori hypotheses (scenarios 1 and 2). Our first hypothesis (scenario 1) was that a pharmaceutical sponsor would favour its own drug in placebo controlled or active comparator trials of statins or the highest dose of its own drug.26 Accordingly, we labelled one arm per trial as potentially biased in the industry sponsored trials of statins. In scenario 2, we hypothesised that a sponsor would favour its own drug in placebo controlled or active comparator trials of statins, but that it would not differentiate between different doses of its own drug in dose comparator trials. Accordingly, we labelled one arm per trial as potentially biased in the industry sponsored trials of statins unless a given trial compared the same statin at different doses.

Statistical analysis

We used a network meta-analytic approach to examine the methodological quality and risk of industry sponsorship bias in the randomised trials of statins. By taking into account actual differences in the effectiveness of individual statins at different doses, network meta-analysis provides an analytic framework for the exploration of bias.23 We evaluated both the methodological quality and risk of bias on the basis of the mean change from baseline in serum LDL cholesterol levels between two comparator treatments for a given dose (change from baseline in the treatment group minus that in the control group).

To determine the dose comparative effects of individual statins, we simultaneously combined the direct within-trial comparisons between two treatments (such as atorvastatin at 10 mg/day v control) with indirect comparisons from trials that had one treatment in common (such as atorvastatin ≤10 mg/day v control and simvastatin ≤10 mg/day v control).27 28 29 When combining direct and indirect evidence, the likelihood for each trial was based on the mean LDL cholesterol change from baseline in each treatment arm and its variance. The information on any multi-arm trial was used in the network meta-analysis just once, thereby correctly informing all treatment contrasts while also correctly taking into account the between-arm correlations in multi-arm trials.30 This analysis preserved the within-trial randomised treatment comparison of each trial while combining all available comparisons between treatments across trials. We combined study-level relative treatment effects using bayesian Markov chain Monte Carlo methods in WinBUGS version 1.4.3.31 We used the model developed by Dias and colleagues for the National Institute of Health and Care Excellence (NICE) Decision Support Unit,30 32 33 assuming that the mean change from baseline in LDL cholesterol per trial arm had a normal distribution. We used the identity function to link the relative effects across trials making different comparisons, while taking into account the correlations between treatment effects within multi-arm trials. All analyses were dose-specific and explored the effects of individual statins at different doses separately. Accordingly, we did not assume any trends and treatment effects at different doses were not “nested” within an overall treatment effect for each statin.

Our models adopted both fixed and random effects for dose comparative treatment effects.34 The fixed effect model assumed that there was no between-study heterogeneity. The random effects model took into account potential heterogeneity by assuming that the trial-specific treatment effects were drawn from a distribution, with a mean that was specific for each treatment, and a common variance shared by all treatments. Our bayesian models adopted non-informative prior distributions for treatment effects (normal (0, 10 000)) and the between-trial variance (uniform (0, 100)) in random effects analyses. We tested the sensitivity of our analyses to prior distributions by examining whether the effect sizes and credibility intervals changed after using more informative priors for the between-trial variance in random effects models (uniform (0, 10)).

To qualitatively evaluate the sensitivity of our base case analyses to study-level methodological quality attributes, we carried out six sets of separate network meta-analyses by excluding studies with low quality on the following methodological items: (1) blinding of patients and investigators; (2) random sequence generation; (3) allocation concealment; (4) blinding of outcome assessment; (5) indications of incomplete outcome data; and (6) indications of selective reporting. We subsequently plotted side by side the treatment effect sizes and 95% credible intervals obtained from each of the six separate analyses and visually inspected any potential discrepancies.

To evaluate potential industry sponsorship bias, we carried out meta-regression analyses.26 35 36 We performed all meta-regression analyses by allowing for a common treatment bias interaction for each statin dose compared with control.37 In addition to assuming that each trial estimated a study-specific bias, which came from a common bias distribution (that is, random effects model for bias), in a separate analysis we also assumed that all studies estimated the same bias parameter (that is, fixed effect model for bias).

To evaluate inconsistency between direct and indirect evidence, we first qualitatively evaluated the consistency of relative treatment effects obtained from an analysis of active comparator trials (that is, direct evidence) with those obtained from an analysis combining both placebo controlled and active comparator trials (that is, mixed evidence). We plotted and visually inspected the relative treatment effects obtained from two sets of analyses (that is, direct and mixed estimates) and found no discernible evidence of inconsistency. As we previously reported,16 there was also no evidence that the potential heterogeneity and inconsistency across the evidence base could be explained by baseline mean age, baseline mean LDL cholesterol concentration, or publication year.

Our analyses employed a long burn-in period (50 000 iterations) and follow-up period (80 000-100 000 iterations) to allow for convergence. We visually inspected trace plots for key parameters for each analysis to assess convergence in terms of stability. We formally evaluated the goodness of fit of our models using the total residual deviance (posterior mean of the deviance under a given model minus the deviance for the saturated model) along with the deviance information criterion (sum of the posterior mean of the residual deviance and the effective number of parameters).32 Because of better fit, we only reported the results obtained from the fixed effect model for bias and random effects model for treatment effects; the total residual deviance for the fixed effect model for bias and random effects model for treatment effects was 228.20 as compared with 229.20 for the random effects model for both bias and treatment effects.