Original Article
The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study

https://doi.org/10.1016/j.jclinepi.2018.12.013Get rights and content

Abstract

Objectives

The aims of the article were to assess the methodological quality of robotic surgical meta-analyses (MAs) using A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2) and to explore the factors of methodological quality.

Study Design and Setting

Robotic surgical MAs published between 2015 and 2018 were identified through a systematical search in PubMed, EMBASE, Cochrane library, and Web of Science databases. The methodological quality of eligible MAs was evaluated by AMSTAR-2. Data extraction and the methodological quality of MAs assessment were double checked by four trained reviewers. The intraclass correlation coefficient (ICC) was used to assess the consistency of quantitative measurements, and the ICC for overall score and score of critical domains were 0.952 and 0.912, respectively. Multivariate regression analysis was used to identify potential factors affecting methodological quality.

Results

A total of 123 MAs focused on 18 surgical locations were included. The findings showed that, regarding quality, only two (1.6%) of 123 MAs were high, two (1.6%) were moderate, two (1.6%) were low, and the remainder 117 (95.1%) were critical low. Multiple linear regression analysis revealed that publishing year and journal rank independently associated with methodological quality of MAs; origin region (P > 0.05), Preferred Reporting Items for Systematic Reviews and Meta-Analyses (P = 0.421), randomized controlled trial enrollment (P = 0.304), and funding support (P = 0.958) did not influence the quality of the MAs. Registration (item 2) and funding reported for individual studies (item 10) showed the poorest adherence in the MAs.

Conclusion

Our study showed that the previously published robotic surgical MAs lack good scientific quality, especially in those published in Q2- to Q4-rated journals. Potential solutions to improve the quality of future robotic surgical MAs include preregistration and funding reported for individual studies.

Introduction

Since being approved by the Food and Drug Administration (FDA) in 2000, robotic surgery (RS) has been widely applied in various surgical fields due to the improved comfort for surgeons and dexterity of movement, minimized hand tremor, and wide range of movements allowing for increased precision [1], [2]. Numerous observational studies [3], [4] and limited randomized controlled trials (RCTs) [5], [6], [7] were performed to explore the advantages of RS compared with laparoscopic surgery (LS), video-assisted surgery (VS), or open surgery (OS) regarding application purposes, whereas the results were controversial [8], [9]. In addition, high-quality meta-analyses (MAs) have been increasingly regarded as one of the key tools for achieving evidence [10], [11]. Therefore, the number of published MAs of comparative studies in RS increase rapidly.

There are risks in uncritically accepting the results of MAs, especially for those MAs with serious heterogeneity or methodological dilemmas [12], [13]. Heterogeneity is inherent in MAs, and not a problem to be solved, whereas methodological dilemmas involve puzzles that are potentially solvable. Therefore, it is important for users to identify high-quality MAs. Although several instruments have been designed to assess the methodological quality of systematic reviews, not all instruments have been systematically developed or empirically validated [14], [15], [16], [17]. For example, A MeaSurement Tool to Assess systematic Reviews (AMSTAR) represented a validated instrument to evaluate systematic reviews of RCTs [15], [18] and has been widely used in many countries to assess the methodological quality of MAs in various fields [19], [20], [21], [22]. However, robotic surgical MAs often include a large proportion of non-RCTs [23], [24].

AMSTAR-2, the updated version of AMSTAR, could be used to assess systematic reviews based on non-RCTs [12]. When compared with risk of bias in systematic reviews (ROBIS) [25], a tool to evaluate risk of bias of systematic review, AMSTAR-2 considered more aspects related to methodological quality, for example, registration and conflict of interest were only included in AMSTAR-2 scale but were related to the quality of systematic review. In addition, AMSTAR-2 provided more specific methods about how to analyze and interpret data than ROBIS.

In this study, we aimed to (1) assess the methodological quality of robotic surgical MAs using AMSTAR-2 and (2) explore potential risk factors associated with the methodological quality of robotic surgical MAs.

Section snippets

Search strategy

We performed a comprehensive literature search in PubMed, EMBASE, Cochrane Library, and Web of Science databases from their inception dates to November 21, 2017. Search terms included the following text words: “meta*analysis*,” “systematic*,” “robotic*,” “Computer* Surgery,” “remote surgery,” “telerobot*,” and MeSH words: “Robotic Surgical Procedures,” “computer assisted surgery,” and “Meta-analysis as Topic.” No limitation was used. The full search strategies are presented in Appendix S1. An

Selection of studies

Through searches of electronic databases, 7,172 records were obtained. After removing duplicate records, 5,223 records were screened, and after review of titles and abstracts, a total of 4,889 records were excluded. Of the remaining 334 records, the full-text was reviewed, and of these, 123 MAs were included in this work. No additional studies were identified through reference review. The study selection flow is detailed by a PRISMA flow diagram and shown in Figure 1.

General characteristics

The 123 robotic surgical

Summary of findings

In this study of robotic surgical MAs published from 2015 to 2018, specific trends in methodological quality were identified, with an increasing improvement over time. However, only two (1.6%) MAs were of high quality, two (1.6%) were of moderate quality, two (1.6%) were of low quality, and the rest 117 (95.1%) were of critical low quality. This indicated suboptimal methodological quality of previously published MAs in the field of RS. The suboptimal quality may be associated with the rank of

Conclusion

Although the number and methodological quality of robotic surgical MAs showed a tendency to improve, the previously published MAs lack good scientific quality, especially those published in Q2- to Q4-rated journals. Reporting of funding sources of individual studies and preregistration may be a potential solution to improve the quality of future MAs. Efforts should be made to improve the methodological quality of MAs in the RS field.

References (50)

  • H. Zhang et al.

    Reporting and methodological qualities of published surgical meta-analyses

    J Clin Epidemiol

    (2016)
  • J. Tian et al.

    The methodological and reporting quality of systematic reviews from China and the USA are similar

    J Clin Epidemiol

    (2017)
  • L. Ge et al.

    Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study

    J Clin Epidemiol

    (2018)
  • P. Tugwell et al.

    Cochrane and Campbell collaborations, and health equity

    Lancet

    (2006)
  • A. Kural et al.

    Can we accomplish better oncological results with robot-assisted radical prostatectomy?

    J Endourol

    (2017)
  • A. Valverde et al.

    Robotic versus laparoscopic rectal resection for sphincter-saving surgery: pathological and short-term outcomes in a single-center analysis of 130 consecutive patients

    Surg Endosc

    (2017)
  • M. Broholm et al.

    Limited evidence for robot-assisted surgery: a systematic review and meta-analysis of randomized controlled trials

    Surg Laparosc Endosc Percutan Tech

    (2016)
  • S. Adie et al.

    Quality of conduct and reporting of meta-analyses of surgical interventions

    Ann Surg

    (2015)
  • V. Ficarra et al.

    Robot-assisted vs open radical prostatectomy: the day after

    BJU Int

    (2017)
  • C.D. Mulrow

    Rationale for systematic reviews

    BMJ

    (1994)
  • D. Young

    Policymakers, experts review evidence-based medicine

    Am J Health Syst Pharm

    (2005)
  • B.J. Shea et al.

    AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both

    BMJ

    (2017)
  • J.A. Berlin et al.

    Meta-analysis as evidence: building a better pyramid

    JAMA

    (2014)
  • B. Shea et al.

    Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools

  • B.J. Shea et al.

    External validation of a measurement tool to assess systematic reviews (AMSTAR)

    PLoS One

    (2007)
  • Cited by (90)

    • Reporting and methodological quality of acupuncture network meta-analyses could be improved: an evidence mapping

      2023, Journal of Clinical Epidemiology
      Citation Excerpt :

      NMA is a statistical method that combines all available evidence for an outcome in several studies and indirectly compares two or more interventions (mainly adjusted indirect comparisons) or the combined outcome of direct and indirect comparisons (mixed treatment effects) to generate pairwise estimates of each intervention to every other intervention within a network [18–20]. However, NMAs are subject to similar methodological risks as standard pairwise systematic reviews [21,22]. Because of their methodological complexity, NMAs may even be more vulnerable to such risks [23–25].

    • Clinical Epidemiology in China series. Paper 3: The methodological and reporting quality of systematic reviews and meta-analyses published by China' researchers in English-language is higher than those published in Chinese-language

      2021, Journal of Clinical Epidemiology
      Citation Excerpt :

      Second, we only included SRs/MAs published between 2016 and 2018. Third, our assessments were only based on published information, so the appraisal results may be limited by underreporting and reporting quality, but this is a common and inevitable disadvantage of this type of research [30]. SRs/MAs conducted by researchers from China published in English or Chinese tended to be of low methodological and reporting quality, and the compliance to many items of the respective checklists was low.

    View all citing articles on Scopus

    Declaration of interest for all authors: none.

    Funding: This work was supported by (1) Fundamental Research Funds for the Central Universities, China (Grant no. 16LZUJBWTD013, Grant no. 18LZUJBWZX006): Evidence-Based Sociology Research; (2) Research Projects of Gansu Provincial Hospital, China (Grant no. 17GSSY9-1, Grant no. 18GSSY3-8); (3) Key Laboratory of Evidence Based Medicine and Knowledge Translation Foundation of Gansu Province, China (Grant no. GSXZYZH2018006); and (4) Laboratory Of Intelligent Medical Engineering of Gansu Province, China (Grant no. GSXZYZH2018001).

    Conflicts of interest: There are no financial or other interests with regard to the submitted article that might be construed as a conflict of interest.

    Authors’ contributions: K.Y. and R.L. contributed to study design, medical/methodological advice, supervising the analysis and interpreting the data, and drafting and revising the manuscript. P.Y. and L.Y. contributed to study design, collecting and cleaning data, analyzing and interpreting the data, and drafting and revising the article. H.L., Y.X., M.L., and C.L. contributed to collecting entering and cleaning data and drafting the article. M.Z., H.C., L.H., and T.G. contributed to interpreting data and drafting and revising the article. All authors were involved in writing the article and had final approval of the submitted.

    1

    These authors are co-first authors.

    View full text