TY - JOUR T1 - Agreement between ranking metrics in network meta-analysis: an empirical study JF - medRxiv DO - 10.1101/2020.02.11.20021055 SP - 2020.02.11.20021055 AU - Virginia Chiocchia AU - Adriani Nikolakopoulou AU - Theodoros Papakonstantinou AU - Matthias Egger AU - Georgia Salanti Y1 - 2020/01/01 UR - http://medrxiv.org/content/early/2020/02/12/2020.02.11.20021055.abstract N2 - Objective To empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement.Design Empirical evaluation from re-analysis of network meta-analyses.Data 232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015.Methods We calculated treatment hierarchies from several ranking metrics: relative treatment effects, probability of producing the best value (pBV) and the surface under the cumulative ranking curve (SUCRA). We estimated the level of agreement between the treatment hierarchies using different measures: Kendall’s τ and Spearman’s ρ correlation; and the Yilmaz τAP and Average Overlap, to give more weight to the top of the rankings. Finally, we assessed how the amount of the information present in a network affects the agreement between treatment hierarchies, using the average variance, the relative range of variance, and the total sample size over the number of interventions of a network.Results Overall, the pairwise agreement was high for all treatment hierarchies obtained by the different ranking metrics. The highest agreement was observed between SUCRA and the relative treatment effect for both correlation and top-weighted measures whose medians were all equal to one. The agreement between rankings decreased for networks with less precise estimates and the hierarchies obtained from pBV appeared to be the most sensitive to large differences in the variance estimates. However, such large differences were rare.Conclusions Different ranking metrics address different treatment hierarchy problems, however they produced similar rankings in the published networks. Researchers reporting NMA results can use the ranking metric they prefer, unless there are imprecise estimates or large imbalances in the variance estimates. In this case treatment hierarchies based on both probabilistic and non-probabilistic ranking metrics should be presented.STRENGTH AND LIMITATIONS OF THIS STUDYTo our knowledge, this is the first empirical study exploring the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA).The study also explores how agreement is influenced by network characteristics.More than 200 published NMAs were re-analysed and three different ranking metrics calculated using both frequentist and Bayesian approaches.Other potential factors not investigated in this study could influence the agreement between hierarchies.Competing Interest StatementAll authors have completed the ICMJE uniform disclosure form and declare: all authors had financial support from the Swiss National Science Foundation for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.Funding StatementThis work was supported by the Swiss National Science Foundation grant/award number 179158.Author DeclarationsAll relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.YesAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe data for the network meta-analyses included in this study are available in the database accessible using the nmadb R package. ER -