Prediction and diagnosis of chronic kidney disease development and progression using machine-learning: protocol for a systematic review and meta-analysis of reporting standards and model performance

Study designs

Any peer-reviewed primary studies which assessed a prediction algorithm that utilizes ML techniques applied to clinical problems in the prediction and diagnosis of chronic kidney disease development and progression, including those for CKD screening, CKD prevention, profiling of biomarkers contributing towards CKD, profiling of risk factors leading to CKD, estimation of glomerular filtration rate (GFR) and creatinine levels, prediction of occurrence of CKD, prediction of CKD stages, CKD diagnosis, CKD prognostication, prediction of CKD progression to ESKD and/or requirement for renal replacement therapy, ESKD diagnosis will be included.

Exclusion criteria are 1) Studies that utilize only image-based inputs as the different model development processes require alternative extraction and appraisal tools; 2) studies assessing prediction models of CKD complications other than its progression, including non-exhaustively anemia, electrolyte disturbances, bone disorders, and cardiovascular events; 3) prediction model of RRT including non-exhaustively the choice of RRT modalities which are hemodialysis, peritoneal dialysis, and renal transplantation, and 4) studies reporting only treatment-related outcomes of CKD such as adverse events, rate of complications, the management of complications; 5) informal publication types such as case studies, commentaries, letters to the editor, editorials, meeting abstracts, proceeding papers, conference abstracts, protocols, guidelines, and recommendations; 6) review articles such as narrative review, overview, systematic review, meta-analysis; 7) animal studies.

Study participants

Adult humans whose age was equal to or more than 18 years old.

Types of interventions

The studies will present prediction models utilizing ML techniques, including non-exhaustively various regression techniques, decision trees, random forests, support vector machines, K-nearest neighbor, and neural networks, as defined by individual studies. The models will be for the prediction and diagnosis of chronic kidney disease development and progression with or without mention of ESKD.

Comparators

We will include studies that compare the performance of ML-based prediction models with those that utilize conventional techniques, including non-exhaustively those that use logistic regression (including penalized LR), cox regression, Poisson regression, least squares linear separation, generalized additive models, discriminant analysis, generalized estimation equations, risk scores, and expert views. Studies that utilize only ML-based tools will also be included.

Primary outcome

• Performance comparison of ML-based and non-ML-based prediction tools in CKD development and progression

Secondary outcomes

• ML-based model use case

• Performance of ML-based prediction tool in CKD development and progression

• Stages of model development (internal or external validation or clinical implementation)

• Model development team specialty and the involvement of model end-user such as clinicians during model development

• Evidence of model reporting quality description

• Characteristics of the dataset (size of training, validation and testing datasets, source of dataset, population group, data period, length of follow-up)

• Prediction model construct including ML-based and non-ML based techniques

• Predictor characteristics and selection

• Model outcome characteristics and selection

• Model performance measures used

Data management

Following database searching, studies will be populated into Covidence systematic review software (29), which will manage study selection and data extraction.

Selection process

We will carry out two stages of screening. After study de-duplication through Covidence, two reviewers will screen the titles and abstracts of potential studies independently. We will eliminate abstracts in the initial screen if they do not report ML-based prediction models in CKD.

Included studies will undergo full-text review against the full eligibility criteria. Reasons for exclusion will be recorded for each study. Disagreement between two reviewers at each article screening and selection stage will be resolved by consensus and a third person if necessary. The PRISMA 2020 flow diagram will be generated to describe the workflow and identification of included studies for the systematic review (30).

Data collection and management

The data extraction form will be designed prospectively before data collection and will be pilot tested and refined. To ensure consistency across reviewers, explicit instructions will be given, and calibration will take place before the extraction process. Two independent reviewers will extract the following data items based on items included in the CHARMS checklist: 1) source of data; 2) participant information; 3) outcome(s) to be predicted; 4) candidate predictors; 5) sample size; 6) Missing data; 7) Model development; 8) Model performance; 9) Model evaluation; 10) Results (final model presented) including model performance; 11) interpretation and discussion. In addition, information will be extracted on the study information (authors, year of publication, study design, journal, contact information, study period, geographical location (area and country), and funding), the assessment of reporting standards using an objective measure if mentioned in the study, and any other relevant information. All relevant text, tables, and figures will be examined for data extraction. Disagreements between the two independent reviewers will be resolved by consensus. We will contact the study authors to request incompletely reported data in included studies. We will conduct analyses using available data if no response is received within 14 days.

Reporting quality assessment

We will assess the reporting quality of studies against the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) statement, which aims to improve the transparent reporting of prediction modeling studies in all medical settings (31).

The TRIPOD statement provides recommendations for reporting studies on developing, validating, or updating a prediction model. To assess the completeness of reporting amongst each publication, we will utilize the published “TRIPOD Adherence Extraction Form,” which evaluates 22 main items deemed essential in evaluating the transparency of prediction model studies. Each article will only be assessed for items and sub-items that it applies to (development, external validation, or incremental value reporting of prediction models) based on guidance from the adherence form. Each TRIPOD item is given adherence elements to help evaluate an item. The presence or lack of an adherence element within the article will be marked down either as a “Yes” or a “No.” For a TRIPOD item to receive a score, all adherence elements must be present. Overall article’s TRIPOD score can be calculated by summing up adhered TRIPOD items and dividing by the total number of applicable TRIPOD items for that article. Findings on reporting quality from the TRIPOD adherence extraction will be summarised and graphically presented.

Risk of bias assessment

We will assess the risk of bias (ROB) of studies by applying PROBAST (prediction model risk of bias assessment tool) (32). PROBAST was developed to assess ROB and applicability concerns of a study that evaluates (e.g., develops or validates) a multivariable diagnostic or prognostic prediction model. Reviewers will assess each study based on the published “PROBAST Assessment Form.” PROBAST is organized into four domains: participants, predictors, outcomes, and analysis. A ROB rating (high, low, or unclear) will be assigned to each domain based on answering signaling questions provided by the PROBAST assessment form. Signaling questions can be answered as yes, probably yes, no, probably no, or no information. Based on ROB results from the four domains, an overall ROB rating and prediction model applicability rating will be given to the prediction model following the recommendations in the PROBAST assessment form. A tabular presentation on PROBAST results for each study will be available. Results will be summarized and graphically presented for each domain.

Comments on Methodology

Two independent reviewers will assess the study quality and risk of bias. To improve review consistency, reviewers will be onboarded on general decision rules and given review practice for TRIPOD and PROBAST before conducting an independent review. Disagreements between two reviewers will be resolved by consensus.

We will contact the author if not enough information is available for assessment. We will utilize the available data if the authors do not respond for 14 days. We will present reporting quality and risk of bias assessment in the respective tables.

Qualitative synthesis

We will provide a qualitative analysis of the studies and their results following standard 4.2 and conduct a qualitative synthesis, chapter 4 of Finding What Works in Health Care: Standards for Systematic Review (33). In addition, we will analyze studies by themes following the primary and secondary outcomes. We will report details of prediction model performance, comparing ML-based and non-ML-based models, the prediction model use case, choice of predictors and outcomes, ML model construct, prediction model reporting standard, and risk of bias.

Quantitative synthesis

The studies will likely show significant clinical, methodological and statistical heterogeneity. We will therefore synthesize data quantitatively in appropriate subgroups (prediction of CKD development, CKD diagnosis, and prediction of CKD progression to ESKD), guided by the Cochrane Handbook of Systematic Reviews of Diagnostic Tests Accuracy, Chapter 10: Understanding meta-analysis. Draft version (13 May 2022) (34).

Measure of effect size

Performance measures of the ML algorithms will be recorded, including 2×2 confusion matrix, sensitivity, specificity, and AUROC or any other measures utilized by individual studies. The most common measure will be selected for further meta-analysis if feasible.

Assessment of heterogeneity

We will assess the clinical heterogeneity of studies based on the ML algorithm use case, the participant characteristics, the predictor choice, and selection. We will assess the methodological heterogeneity based on the ML algorithm construct regarding the data size, ML technique, and performance measures. We will assess statistical heterogeneity using the χ² test and the I² statistic. We will consider an I² value greater than 50% indicative of substantial heterogeneity.

Quantitative Data Synthesis

Suppose sensitivity, specificity, 2×2 confusion matrix results are available for the majority of studies, we will use either a bivariate model or Rutter and Gatsonis HSROC models depending on the consistency of study thresholds to calculate to define a summary point or SROC curve of the algorithm performance estimate combining sensitivity and specificity.

We will undergo exploratory analysis comparing the performance of ML and non-ML techniques. Depending on the number of studies that evaluated both methods, we will either restrict the analysis to these studies or combine all studies that have utilized one of the techniques. We will again use the bivariate model or the Rutter and Gatsonis HSROC model to compare the relative accuracy of both tests.

If the majority of sensitivity, specificity, and 2×2 confusion data are not available, and if AUROC is the most common performance measure, we will utilize methods described in Christodoulou et al. (35) by analyzing pairwise differences in logit AUROCs between ML-based and non-ML based techniques by random effects modeling by DerSimonian and Laird method, either pooled or within subgroups stratified by the risk of bias, study outcomes, ML techniques. Points and the 95% CI will describe the pairwise differences in logit AUROCs.

The meta-analysis will be performed using Review Manager version 5.4.1 (The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark) (36) and Stata (37).

Additional analysis

Further subgroup analyses will be performed to explore possible sources of heterogeneity based on the following: study quality (risk of bias and reporting quality assessment), ML techniques, stages of validation, and dataset (stratified as determined by resulted studies).

We will conduct sensitivity analyses based on study quality, study publication years (stratified by year), study populations, the ML model construct, the ML user case, or any other relevant strata.