Introduction

The advent of whole slide digital imaging systems has catalysed new interactive models for pathology education, expedited clinical workflows and provided opportunities for the development of new medically actionable image analysis tools and investigative techniques. Although still evolving, this technology has provided the clinical and research community with new solutions that will need to be integrated with established models for contemporary care. Ultimately, we anticipate that a new system of care will emerge, whereby these innovative technologies are integrated into routine practice in a rational and fully validated manner.

In this Review, we discuss how developments in digital pathology and computational image analysis are shaping a new digital era for nephropathology and how these technologies might be most expeditiously considered for clinical deployment. We provide an overview of the current status of the digital ecosystem in nephropathology, which encompasses human expertise in renal pathology, artificial intelligence (AI), omics science and powerful computational tools, as well as potential applications and challenges associated with the emerging armamentarium of technologies for image analysis. Finally, we present what might be a prescient vision of the imminent future of clinical research and practice made possible by these technologies.

Nephropathology in the digital era

Only a decade ago, virtual microscopy was considered to be a fairly novel technology and of insufficient maturity for primary diagnostic purposes. Although the technology was rapidly embraced for educational uses, its role in clinical practice was less clear given the preference of conventionally trained pathologists to work with conventional light microscopy. This status quo has been shattered by the emergence of rapidly evolving technologies such as telepathology and digital pathology, in combination with developments in computational image analysis. The collective subspecialties of pathology are now witnessing a far-ranging and transformative set of new practice models, with implications for both intramural and remote primary and consultative diagnosis. Similarly, delivery models for education and companion diagnostics are positively affected by the emergence of these technologies. In particular, content-based image retrieval1 — a machine vision technique that searches images by analysing their content (according to their colour, shape and/or texture) rather than their associated metadata — is now a reality that opens fascinating new possibilities to leverage the vast and under-utilized slide and image archives long held by pathology departments. The main uses of digital imaging in nephropathology can be divided into three major operational models: telepathology, digital pathology and computational image analysis, as outlined below.

Telepathology

Telepathology involves the transmission of one or more microscopic images from an originating location to an imaging workstation at a distant location. Its use has been extensively validated and it is now a common tool for real-time assessment of tissue adequacy and diagnosis2,3. In nephropathology, dynamic telepathology has been deployed for the remote assessment of donor implant biopsy samples by trained renal pathologists. This approach overcomes some of the limitations of current practice, whereby cadaveric donor kidneys are often harvested and evaluated after-hours, in non-academic hospitals and by general surgical pathologists4,5. Telepathology has also enabled renal pathologists to assess the quality of fresh tissue biopsy at the time of the biopsy procedure without needing to be physically present in the interventional radiology suite. Its use gives the subject matter expert (rather than the available non-renal pathologist) the ability to triage the kidney biopsy samples for subsequent histology, immunofluorescence and electron microscopy analysis, without significant disruption of their workflow1.

Digital pathology

Digital pathology is a general term that refers to the assemblage of digital workflow and imaging solutions that are geared towards creating a digital image-based practice environment in which a whole slide image (WSI) or other digital image is acquired, managed, interpreted and searched for specific content. This new approach is being increasingly adopted in clinical trials6, for education purposes7,8,9 and for clinical research1,10,11,12,13,14,15,16. Since 2017, the FDA has approved the use of commercial WSI platforms, facilitating the use of digital pathology as a tool for primary diagnosis17,18,19,20,21,22,22. In response, many pathology practices and laboratories, particularly in Europe and Canada, are now in the early or mid-phases of implementing digital pathology solutions for purposes other than image archiving or for second-opinion consultation23,24. WSIs, which can now be scanned in less than a minute, can serve as an effective surrogate for traditional microscopy-based pathology20,25. However, WSI workflow models have known limitations including the need for substantial network bandwidth to handle large file sizes, the lack of Z-dimensional focus (except in specialized cases, where a ‘Z-Stack’ image might be generated for a region of interest (ROI) from the larger total slide area), which limits their utility for cytopathological and renal biopsy evaluation, and imperfect control software in the scanning appliances themselves, which can contribute to various imaging artefacts (for example, out-of-focal plane dust on the glass slide triggering acquisition of the wrong focal plane)26.

These limitations support the subjective perception held by many pathologists that digital pathology can be less accurate and more time consuming than glass slide microscopy for diagnosis, although direct quantitative comparisons show the opposite. Processes might indeed be slower when digital pathology workflows are first adopted; however, following an initial training and acclimatization period, overall pathologist performance greatly increases for measurements of both speed and accuracy. As a result, perceptions are changing, with digital pathology becoming increasingly accepted, helping the pathology specialty move towards fully digital workflows and associated platforms for clinical operations26,27,28,29.

Computational image analysis

Advances in scanning technology paired with the increasing availability of large datasets of digital images have enabled pathologists to collaborate with technology experts, including data scientists, computational engineers and imaging physicists, to explore the potential of a newly established branch in the field of pathology: machine vision, a component of computational image analysis. The primary objective of this field is to extract and/or generate quantitative data from digital image subject matter, either in isolation or in tandem with other classes of spatially or non-spatially based biological or omics data. Recognizing the high-dimensionality of the data generated by computational image analysis, AI and machine-learning (ML)30,31,32,33 techniques, which are ideally positioned to interrogate such complex datasets in an exhaustive manner, have been deployed to extract features, patterns and information from histopathological subject matter that cannot otherwise be analysed by human-based image interrogation alone. Deep learning (DL) — also known as deep structured learning — is a particularly important branch of ML that provides opportunities to interrogate images at greater depths than previously possible30,31,34,35. In pathology, DL has been used for the detection, annotation, segmentation, registration, processing and classification of WSIs. The learning process uses neural networks mimicking the human intelligence and contains multiple layers to progressively extract deep features from the input image, first transcribing imaging information into structured data, and then translating them into knowledge.

AI models, including ML and DL, can be used for various purposes, including the automatic detection, segmentation and quantification of histological parameters and structural changes, and for disease diagnosis. These tasks are considered to be ‘low-level’ tasks. This is in contrast to ‘high-level tasks’, in which AI models are used to concurrently interrogate and integrate multiple classes of primary data — for example, histopathological image data spatially coupled to transcriptomic data — providing opportunities for the prediction of processes such as disease aggression (for example, the biological potential of a malignancy), patient outcome, organ engraftment survival and therapeutic response. Thus, use of AI has the potential to elevate digital images from their basic use as a tool for visual assessment of disease status to a much more complex and comprehensive role as a tool to facilitate prediction of disease trajectory. These new capabilities enable the extraction of useful information in a way that was not previously possible with the use of conventional microscopy in the context of direct human cognitive and visual assessment.

The increased demand for predictive assays for patient profiling is driven not only by the need to better stratify patients according to their clinical characteristics and outcomes but also by the evolving discovery and development of drugs that target specific molecular pathways36. Novel approaches to aid in the understanding of the relationships between structural changes in tissue and clinical and/or molecular phenotypes would be particularly beneficial to the nephrology community, as current classification systems are inadequate in their ability to capture the heterogeneity of kidney diseases, or predict disease outcomes. New, robust algorithms derived from the application of AI approaches and integration of pathology data with other relevant datasets will have utility for both pathologists and nephrologists, with the potential for a meaningful impact on patient care and precision medicine.

Many parallels can be drawn between machine vision in pathology and developments in radiography, which underwent its own digital revolution in the late twentieth century with the development of native digital radiographic modalities such as MRI and CT, which allowed interrogation of digital anatomically coupled subject matter for the first time. Additional innovations in radiology, such as cassette-based storage phosphor image plates (which allowed X-ray images to be recorded digitally37) and the implementation of charge-coupled devices, facilitated the rapid development of digital detectors38,39,40,41 and further accelerated the adoption of digital technologies. Similar to the current interplay between digital and computational image analysis, the mainstream availability of digital radiographic images has enabled quantitative imaging analysis techniques to be tested and introduced for use in clinical trials and for patient monitoring42,43.

Digital pathology repositories

The nephrology community has witnessed the formation of several international consortia that collect biospecimens from patients with a variety of kidney diseases (Table 1). The common goals of these consortia10,11,13,44,45,46,47,48,49,50,51,52 are to better understand the pathogenesis of kidney diseases and to improve their classification and treatment through comprehensive analysis of clinical, morphological and molecular data — a process that has required the establishment of digital pathology repositories (DPRs) for the banking and organization of digital images from renal biopsies and their associated metadata.

Table 1 Multicentre digital pathology repositories for kidney disease
Full size table

The establishment of DPRs has driven the concept of a digital biobank and a convergence of research on digital rather than physical assets. Conceptually, the formation of a digital biobank is the same as forming other types of biobank, starting with rigorous training of personnel involved in the collection, de-identification and transfer of the pathology material. This part of the process requires adherence to quality control (QC) steps to protect the privacy of individuals and their health information in accordance with the Health Insurance Portability and Accountability Act, to ensure study protocol compliance and that the renal biopsy pathology materials are adequate, complete, organized in the appropriate patient and disease category-designated space, and properly identified (labelled) for searchability11,14. DPRs are created to facilitate shareable data access; thus, considerations are needed at the consortium and regulatory agency level to facilitate secure, seamless and timely access for the end user.

The digital nephropathology ecosystem

Digital pathology applications are progressively modifying the ecosystem in which pathologists operate, offering considerable advantages over traditional pathology approaches, as discussed below. However, these applications also present challenges that require consideration of factors such as the redistribution of resources, standardization and regulation of processes, approaches to standardizing and integrating data, the consequences of these new approaches for diagnostic paradigms and effects on the medical curriculum.

Investment in digital pathology

DPRs are a cost-efficient way of systematically organizing pathology resources. The investment, although initially burdensome, is long term and allows for the establishment of a permanent library of sharable images.

In translational research, the return of such investment resides in the nature of the digital library itself, which can be accessed by multiple investigators, thereby supporting and facilitating a variety of pathology-based applications and related new discoveries within and across populations15,16,53,54,55,56. Additionally, DPRs allow barriers that were previously not addressable to be overcome: they provide transparency for regulatory agencies and enable collaboration both within and between consortia. The reproducibility of observations is also improved, facilitated by digital annotation11,12,15,16 and computer-aided quantitative assessment, which offers a numerical value to the observations and measurements of parameters (i.e. density, spatial distribution and relationships of histological primitives) that cannot be achieved by conventional analysis1. Thus, with the increasing role of big data and the ability to integrate separate knowledge domains (pathology, clinical and molecular domains) to generate new diagnostic categories and predictors of outcome, the launching of digital platforms for the collection and analysis of tissue structural changes has become a necessary investment for clinical research.

Improvements in image resolution and digital scanner capacity, as well as the development and regulatory approval of whole slide scanners for primary diagnosis and tools for the integration of image management software with laboratory information systems, have positioned digital pathology to be deployed for clinical applications17,19,28. Studies have shown that the implementation of digital pathology in clinical settings decreases overall operational costs while increasing clinical and operational efficiency28. Most recently, the COVID-19 pandemic has forced the practising pathology community to explore approaches to enabling remote histopathological analysis on a large scale. In the USA, a critical step in facilitating this move was an amendment to the 1988 Clinical Laboratory Improvement Act (CLIA ’88), which relaxed the location-specific licensing requirements that regulate where a pathologist is allowed to diagnose cases. This change enabled a number of pathology practices and health networks to rapidly employ remote diagnosis workflow models, allowed the field of pathology to gain greater proficiency in rendering primary diagnoses by digital means and facilitated the accelerated adoption of modality-specific workflow models and digital consultative review. These developments all bode well for the prospect of nephropathology benefiting from the expanded use of WSI solutions for expedited primary diagnosis.

However, the potential benefit of a fully digital pathology operation on clinical applications in nephropathology extends beyond operational aspects alone. Integration of knowledge generated by these new computational tools can lead to new discoveries with the potential to introduce new paradigms in clinical practice.

Standardization of analytics

The deployment of digital pathology into research, clinical trials, and ultimately clinical practice depends on a variety of elements, from the standardization of analytics to allow the sharing of images to the application of image analysis tools (Table 2). The process of transforming glass slides into WSIs involves a series of steps, each contributing to the final quality of the digital image that is visualized on a computer monitor. Protocols for the processing of tissue into histology sections and the scanning and transmission of image-associated information need to be standardized so that the digital pathology applications can augment the work of the pathologist on a large scale. Such processes demand a revision of traditional criteria for QC and standardization of analytics beyond current recommendations and regulations that apply to conventional histology approaches, as provided by the College of American Pathologists, CLIA ‘88, the Digital Pathology Association57 and the Renal Pathology Society (RPS). In 2018, The Royal College of Pathologists in the UK published best-practice recommendations for implementing digital pathology, highlighting the diagnostic challenges of WSIs and providing practical solutions for reducing the risk of discordant diagnoses58,59. In the past few years, the College of American Pathologists has also provided guidelines for validating digital pathology approaches60, and the National Society for Histotechnology has initiated a programme to improve the quality of WSIs60,61. However, we and others recommend that all stake-holders participate in regulating and monitoring the use and performance of digital pathology-based protocols62.

Table 2 Current limitations and proposed solutions for computer-based image analysis
Full size table

As outlined below, standardization of pathology analytics requires consideration of three major components (Fig. 1): a pre-analytical phase, which includes the steps from tissue procurement to fixation, processing, cutting and drying time; an analytical phase, which includes a histology phase (involving stain selection, optimization and validation) and a digital phase (scanning, consideration of image and monitor resolution, number of colours, colour distribution, compression ratio and image format63); and a post-analytical phase, which involves the analysis and interpretation of results, the recording and reporting of data, and pathologist and ML performance metrics1,10,12,16,64,65,66 (Fig. 1).

Fig. 1: The analytical phases of digital pathology.
figure 1

The pre-analytical phase includes steps involved in tissue procurement, processing and fixation. The analytical phase involves a histology phase (which includes selection of the stain to be used, optimization and validation of the staining procedure) and a digital phase (which includes scanning the slides as whole slide images (WSIs) and curation of the digital library). The post-analytical phase involves data extraction, analysis and interpretation of results. Data extraction is accomplished using human visual analysis and/or machine vision, using human–machine synergistic protocols. Data extracted from the images using human and machine vision are integrated and reported. Data integration can be achieved using computational tools as well as human intuition and domain expertise.

Full size image

Standardization of the pre-analytical and analytical phases

The current criteria for ensuring QC of the histology preparations allow for broad variation of pre-analytical and analytical artefacts across and within laboratories. The spectrum of artefacts varies in severity and quality. Some can affect the glass slide (for example, pen marks, dirt and bubbles), whereas others affect the tissue (for example, the presence of tissue folds, or differences in thickness or stain intensity) or the scanning process (for example, differences in focus or the gridding effect of the WSI). While pathologists train themselves to make an interpretation using the stain protocol they are accustomed to and read through artefacts on the glass slide, tissue section or digital image, computers must be trained to adapt to such heterogeneity in image presentation. For computer-aided algorithms to leverage digital pathology and enable quantification, classification and prognostication in a scalable manner, the algorithms must be generalizable across datasets. Although variability in WSI training sets may actually represent a potential advantage for the generalizability of the resulting algorithms, the performance of algorithms is intimately linked to the control of analytics and homogeneity of the datasets63.

High-quality datasets can be achieved by proactively gaining tighter control over analytical practices within and across laboratories, and by re-modelling current criteria for the minimum QC threshold based on the needs of current technologies and their applications61. As the use of image-based data by AI technologies requires the conversion of analogue data into numerical values (that is, digitization), understanding of the intimate association between stain quality and image quality becomes essential. Similar to the field of radiology, where imaging physicists are employed to ensure QC of clinical images and imaging systems, standardization and quality assurance at both the tissue staining phase and the image acquisition phase will likely require on-site technical expertise. This support will ensure patient safety and reliable diagnostics. We expect that this area of optics and light microscopy will be an important focus of discovery and standardization in the coming years.

One relevant issue that requires better understanding before computational tools are deployed for image analysis in the clinical setting is how variation in image quality caused by the physical abstraction layer of the WSI scanning platform itself affects the performance of AI approaches. Different commercial WSI solutions digitally reproduce colours based on a reference colour source slide with uneven variability. This variability can extend to inter-device variability, even within a single model of a WSI scanner. Although colour correction software can be used to standardize WSIs across different scanning platforms, this approach is still in its infancy. Recognizing that AI algorithms can be (and often are) dependent on consistent colour representation, standardization of colour becomes essential to establish robust and reproducible quantitative measurements and analysis63,67,68. This issue is of critical importance in the context of multicentre DPRs and will force investigators to weigh the pros and cons of central versus local scanning.

QC of image datasets can also be achieved retrospectively (that is, after WSI acquisition) using computational post-processing techniques. QC tools have now been developed to automatically assess WSI quality, capturing variations in image presentation such as stain distribution, brightness, or glass, tissue and scanning artefacts. One example of such a QC algorithm is HistoQC, an open source QC tool that quantitatively assesses the heterogeneity of WSI image datasets and, similarly, identifies artefacts present on the source glass slides or on the scanned image69. In the context of clinical research, HistoQC is currently being evaluated in NEPTUNE, CureGN and Kidney Precision Medicine Project (KPMP) for curation of the image dataset prior to experimental analysis (Table 1). In clinical practice, the ability to identify compromised WSIs before they are assessed by the pathologist or computer-aided tools reduces downstream delays in clinical diagnosis.

Standardization of the post-analytical phase

The digital ecosystem has the potential to facilitate the harmonization of data collection both within and between consortia via the use of shared protocols10, and to improve accuracy15 and concordance among pathologists12,16. However, agreement among nephropathologists in diagnostic and tissue assessment remains the limiting factor and a pre-requisite for the training of AI algorithms5,12,16,70,71,72,73. Awareness that the standardization of pathology definitions and use of a controlled vocabulary are central elements that drive the reproducibility and accuracy of observations and/or of data generated for both human and computer-aided analysis is reflected by several recent initiatives. For example, NEPTUNE and the international digital nephropathology network, INTEGRATE, have generated a list of individual observational lesions (descriptors) and their associated definitions, obtained by international consensus, for the assessment of WSIs10,12. Similarly, the Renal Pathology Society has undertaken an initiative to develop a universal lexicon by compiling detailed and unambiguous definitions of pathological lesions and structures with the goal of harmonizing classifications, whereas the KPMP investigators have applied a controlled vocabulary to assemble libraries of conventional diagnoses, disease categories and descriptors. In KPMP, the descriptor library will be relevant to informing and guiding the assembly of the KPMP feature vector library, a ML approach to the automatic detection of normal and abnormal histological primitives, and to generate a functional ontology and atlas of kidney diseases.

Shifting paradigms in renal pathology

The transformative approach to the study of kidney diseases introduced by systems biology science is calling for a change of paradigms in renal pathology for the interrogation of WSIs. The current ‘one-size-fits-all’ approach, whereby pathologists generate classifications using a few preselected parameters and test them against disease outcome or disease biomarkers, provides neither sufficient granularity nor a quantitative metric that adequately represents the complexity of structural and cellular changes that can exist in the kidney. The new digital ecosystem and the development of novel technologies offers an opportunity to establish new protocols and methodologies — either visual or computer-aided — to retrieve and represent, as interpretable data, the information contained in kidney biopsy samples.

The introduction of digital pathology and use of descriptors to identify discrete elements of structural changes in tissue has enabled a more comprehensive visual assessment of the kidney on annotated WSIs compared with conventional methodologies10,11. As discussed above, the standardization of pathology vocabulary is a prerequisite for the interoperability of image data collection and reporting. Use of standardized vocabulary and DPR protocols has facilitated the harmonization of data collected across multiple consortia, providing insights into the structural manifestations of kidney diseases within and across populations10,53. For example, one study that used visual assessment of samples from three cohorts in North America and China showed that the age-adjusted percentage of global glomerulosclerosis is a useful predictor of kidney disease progression across diseases and populations53. Similarly, the percentage of interstitial fibrosis was associated with the expression of specific genes, transcriptional regulators and urinary biomarkers across cohorts from North America and Europe54,55. However, visual (that is, manual) assessment of WSIs is time consuming and has limited reproducibility71. Computational imaging algorithms are likely to provide more efficient and reproducible automated quantitative assessment across diseases and populations, increasing the robustness of the observations and scalability of such studies. The integration of computational image analysis and AI into the digital pathology ecosystem will induce further shifts in renal pathology paradigms towards the integration of structural changes, with multiaxial datasets to redefine categories and subcategories of patients with similar biological mechanisms, morphological profiles and clinical trajectories.

Integration and interoperability

Multi-domain datasets generated by systems biology approaches are represented by different length scales; for example, serum creatinine is measured quantitatively at different time points during the disease course, morphology data are collected using different methodologies, human or machine vision, and metrics, at the time of the biopsy, genetic data do not change with time, whereas the expression of genes and associated regulators are dependent on disease status. The extraction, alignment, integration, interpretation and reporting of such multi-scale information is an emerging challenge in nephrology (Table 2). In the new data integration ecosystem, foundational, structural and semantic interoperability across and within datasets and studies is intimately linked to the standardization of analytics, data sharing and data integration approaches. The interoperability of image analysis alone carries challenges, from annotation to algorithms and image features. For example, image annotations created by vendor-supplied software programs in one community are often incompatible with those created by other communities who might make use of a different program. A valuable approach to facilitating the interoperability of heterogeneous data within and across domains is to use ontologies, which encompass and represent properties and relations between data, categories, concepts and entities, and to organize the data into information and knowledge74,75. The Quantitative Histopathology Image Ontology is one example of an effort to establish standardized approaches to ontological pathology image data for their integration with data from other domains — such as clinical or demographic data. The Quantitative Histopathology Image Ontology is aimed at standardizing terminology at multiple levels, including data input and output, parameters for image acquisition and analysis, and the execution phase whereby data are recorded and reported76. Integration and interoperability are also centre-stage concepts in KPMP, where the development of a kidney cell atlas will be based on a comprehensive ontology harmonized with pre-existing ontologies. Much can be learned by studying other disciplines, such as oncological pathology42,77,78,79,80,81,82,83,84,85 and radiology, where considerable progress in integrating information has already been achieved42, and where ML approaches have been used to combine pathology and molecular data for improved prediction of cancer recurrence81,82,86.

The data ecosystem

The confluence of large-scale studies, often involving clinical data from electronic health records or data repositories, and the capacity to represent entire research studies in electronic form, has altered the way in which researchers collect, store, use and integrate data. The oncology field has integrated data from different domains for quite some time, with the advent of secure data environments paired with flexible database design and cloud computing technology aiding the evolution of the data ecosystem. Rather than attempt to place data in a single database in which a single framework unifies disparate data types, the trend is to now store data in individually optimized databases that are then connected by data aggregators. Functionally, such a set-up enables end-users to perform a search from one database as simultaneous queries against multiple databases. Moreover, this process enables image data to be linked with both clinical and molecular data.

Data sharing

Data sharing encompasses several diverse concepts, from transparency and regulation, to interoperability (discussed above). A key aspirational goal for data sharing has been, and remains the attainment of interoperability, under the umbrella of the now-recognized FAIR principle (Findable, Accessible, Interoperable and Reusable)87. The past decade has seen the emergence of several policies relating to data sharing and research transparency by the Office of Research Integrity, National Institute of Health or other organizations88. Providing access to bio-specimen repositories and research data increases their potential utility for health discovery and validation, and optimizes their long-term value89,90,91,92,93,94,95. This benefit is particularly relevant in the context of rare diseases, where data sharing is often needed to obtain sufficient sample sizes. Historically, renal pathologists have used local datasets for research purposes, with limited opportunities for external QC or data exchange. The availability of digital WSIs fosters opportunities for collaboration in a transparent environment. Transparency is not only relevant to QC for internal research purposes, for example, when clinical trials are conducted using a digital pathology platform for the pathology analysis, the relevant pathology data elements can be easily audited by regulatory agencies6.

However, the concept of data sharing raises several legal and ethical issues. Some of these, including issues related to inclusiveness, ethically responsible research, privacy and return of results, are not new and apply to many forms of research. Consideration must also be given to the sharing of metadata, including validated feature detector ensembles and associated image sets that would be submitted in the curation process in parallel with the bio-specimens themselves. In addition, although best practices relating to data sharing and material transfer agreements have been articulated in many documents, particularly in the context of rare disease research89,90,91,92,93,94,95, guidance specific to the sharing of pathology images and use of DPRs does not yet exist and is urgently needed (Table 2).

An additional consideration is the use of WSI data for retrospective analysis, beyond the original stated research intent. The digital permanent image libraries that will originate at the institutional level, or under individual consortium oversight, will naturally outlive the scope of the originally stated informed consent, the specific aims of the original study, and technology that was available at the time of the image data collection. Given that new technologies empower investigators to generate hypotheses and ask scientific questions that were not conceivable at the time of the collection, it will be essential to establish standard processes to enable permission lists of original data use to be extended. These extensions will require a standardized adjudicative process that can operate without the need to re-consent participants, as their location and identity may no longer be known. Such broad consent documents, which essentially ‘future-proof’ possible unforeseen uses of curated tissues and images are already used by researchers in various omics fields. One controversy from the oncology field96, involving the alleged exclusive sharing of pathology datasets originally intended for general, not-for-profit use with a commercial AI developer, underscores the need for the pathology field to rethink not only how informed consents should be formulated but also who should benefit from the sharing of digital datasets and associated health information.

It is imperative that guidelines for the responsible sharing of data in the digital pathology ecosystem are formally established. Such guidelines could be modelled on previously published documents, such as the international code of conduct for genomic and health-related data sharing, the founding principles of which are based on the goals of promoting health and advancing research, and associated ethical elements (for example, respect of the parties involved and their privacy, distribution of benefits, trust, integrity, accountability, reciprocity, and data security and quality)97.

Image analysis

Two main approaches are used for the interrogation of WSIs: human visual assessment and machine vision.

Human visual assessment

A variety of methods have been used for the visual assessment of WSIs, including conventional diagnostic5 and descriptor-based approaches10, quantification of structural and cellular elements6,12, and morphometric analysis56. The collection and quantification of visual data can be further facilitated by digital or manual annotation tools of the regions or objects of interest on the WSIs, resulting in increased accuracy and reproducibility of the observations15. Visual assessment of WSIs has been used to identify features that correlate with outcomes or the presence of biomarkers, and for the discovery of clinically relevant disease categories. As mentioned earlier, features such as extent of age-adjusted global glomerulosclerosis or interstitial fibrosis can be reliably assessed visually and are associated with outcomes including disease progression53,54. Other quantitative information that can be obtained from WSIs of renal biopsy samples, such as the interstitial capillary and podocyte density, might enhance our ability to predict predisposition to chronic kidney disease progression, hypertension98 and the development of global glomerulosclerosis99,100,101,102,103,104,105 but cannot be easily visually assessed, at least not without a significant amount of labour.

Machine vision

Machine vision refers to the computational techniques that are used to convert digital pathology images into mineable data, using AI techniques such as ML, DL and pathomics. This approach is driving a fundamental paradigm shift in digital pathology in which digital renal biopsy samples are interpreted as quantitative datasets. The performance of the AI techniques depends on the quantity and quality of the primary data, where ‘quality’ refers to the cleanliness, provenance, accuracy, signal-to-noise ratio and comprehensiveness to achieve the maximum predictive performance. Obtaining primary data of sufficient quality or quantity can be challenging for datasets associated with rare kidney diseases.

The two main categories of machine vision that are applicable to WSIs of renal biopsy samples are supervised learning and unsupervised learning. In unsupervised machine vision, features are identified based on their similarity to each other and group or category assignment is based on their relative proximity. By contrast, supervised learning uses an annotated training dataset to learn the relationship between individual features and categorical labels. Supervised and unsupervised learning can be used to convert digital pathology images into minable data, using two general techniques: ‘hand-crafted pathomics’ (also known as conventional pathomics) and ‘discovery pathomics with DL’.

Hand-crafted pathomics

This approach involves the segmentation (by manual or automated means) of an ROI on an image (for example, glomeruli, tubules, arteries or interstitial capillaries), from which high-throughput quantitative features known as hand-crafted features are extracted to derive a pathomic feature space, or digital image signature. This approach is essentially a transcription process, whereby unstructured imaging data (that is, pixels within the ROI) are converted into structured imaging data (that is, the pathomic feature space) so that it can be recognized by learning algorithms.

Pathomic signatures, which are digital imaging signatures obtained through hand-crafted pathomic feature extraction, are typically engineered to capture a number of features including morphology, intensity, texture, higher-order and spatial features. Morphology features quantify the size and shape of the ROI, such as its compactness, border irregularity or maximum diameter. Intensity features capture the distribution of pixel values within the ROI, based on the image’s histogram, which is a probability distribution that encodes the frequency of occurrence of all pixel intensities. Of note, intensity features do not capture spatially encoded information. For instance, the entropy of an image can explain the degree of randomness in its pixel intensities but not where the randomness occurs. Texture features quantify the spatial distribution of pixel information in the image, based on joint-probability distribution functions. These features quantify aspects such as overall image brightness, local heterogeneities and non-linear relationships between pixels. Fine- and coarse-texture features capture heterogeneities at different orders of magnitude. Higher-order features capture information specific to certain frequency bandwidths such as wavelet decomposition and multi-scale invariance such as fractal geometry. Spatial features capture the spatial organization or architecture of specific histological primitives (for example, glomeruli or tubules) in the tissue. These typically involve graph network-based algorithms such as the Voronoi tessellation or minimum spanning tree.

Discovery pathomics with deep learning

In contrast to hand-crafted pathomics, discovery pathomics uses DL to extract features directly from images, without explicit mathematical definitions. DL algorithms are often able to uncover novel and important patterns from sets of input images with case-level annotations and without human guidance, given their ability to excel in unsupervised feature extraction. This approach can be particularly powerful for applications that remain as unstructured imaging data (for example, generating segmentations of a histological primitive), or when a problem is too complex for conventional feature engineering. Although DL can be interpreted as a complete classifier, it can also be used as a type of image representation analogous to the hand-crafted pathomic approach. In fact, DL approaches commonly encode an input signal into structured data form (that is, a transcription process similar to that used for feature engineering), followed by a decoding of the data to generate an output signal (that is, a translation process akin to that used by conventional ML algorithms). The most common type of DL used with images is based on convolutional neural networks (CNNs), whereby a series of shift-invariant filters are used to generate abstract feature maps to learn hierarchical patterns embedded within the input image. CNN architectures have been used to automatically detect and segment normal and sclerotic glomeruli, nodular glomerulosclerosis, tubules, interstitial space and fibrosis, peritubular capillaries and arteries72,73,106 (Fig. 2). Other popular DL architectures (that is, artificial neural networks) include fully convolutional networks, recurrent neural networks and generative adversarial networks42 (Table 3).

Fig. 2: Machine vision technology as a support tool in nephropathology.
figure 2

Machine vision tools — including discovery pathomics with deep learning and hand-crafted pathomics — can be used to convert digital pathology images into minable data and provide support to pathologists. a | Machine vision can be used to automatically detect histological primitives. For example, a convolutional neural network can be used to detect glomeruli in frozen kidney sections stained with haematoxylin and eosin. b | Machine vision has also been used to establish a classifier for glomerular disease stage in patients with diabetic glomerulosclerosis. The left panel shows a paraffin-embedded section of a normal glomerulus (top) and a glomerulus with nodular glomerulosclerosis (bottom) stained with periodic acid Schiff. The middle panel shows detection of the cellular/nuclei (blue) and matrix (red) component of the normal (top) and diseased (bottom) glomerulus. The right panel shows the measurements of the glomerular characteristics for both the normal and the diseased glomerulus. c | Application of artificial intelligence-guided morphometry can provide quantitative assessment of the interstitial fractional space. On the left is a paraffin-embedded section stained with trichrome. The right panel shows the automatic detection of interstitial fractional space. The superimposition of a digital grid on the image enables digital morphometry. d | Machine vision can also be used to build models to aid prognostication. For example, qualitative and quantitative automatic detection of features of acute tubular injury may predict the course of the disease and response to therapy: the presence of only a few areas of vacuolization with specific qualitative characteristics could predict rapid recovery from an episode of acute renal failure, with normalization of serum creatinine levels, compared with a renal biopsy containing much greater levels of vacuolization. e | Computational imaging tools can be combined with other methodologies for parallel discovery. For example, computational image analysis tools can be applied to guide laser capture microdissection by identifying structures with similar pathomic signatures within the same biopsy sample and across biopsy samples. The structures with similar pathomic signatures can be captured and analysed separately, allowing for spatial mapping of pathogenomic signatures. Panel b reprinted courtesy of P. Sander and B. Ginley, University at Buffalo, NY, USA. Panel c reprinted courtesy of J. Hodgin, University of Michigan, MI, USA.

Full size image
Table 3 Use of methods for image analytics in digital nephrology studies
Full size table

Trade-off between hand-crafted and discovery pathomics

Hand-crafted pathomics and discovery pathomics are complementary approaches to quantitative WSI representation. Each technique has advantages and disadvantages, and their proper implementation therefore depends on the specific clinical case and intended use. Hand-crafted pathomic features are mathematically well defined, interpretable and can be easily combined with other structured health data, such as radiomics, transcriptomic or electronic health record data, to develop multi-modal digital biomarkers. However, hand-crafted features are limited to human intuition (that is, human involvement is required to define the features based on relevant domain knowledge) and can often lead to high-dimensional feature spaces that are non-trivial and computationally expensive to analyse. On the other hand, unsupervised feature generation does not require classical feature engineering (that is, features are instead generated automatically based on intrinsic patterns in the data that do not require direct human input). Furthermore, DL-based approaches are particularly useful when the input (for example, manual segmentation of a histological primitive) and output (for example, automatic segmentation) of a problem are both images or when conventional approaches cannot perform the task at hand. However, the trade-off of these benefits is loss of mathematical interpretation and the need for relatively larger datasets.

Applications and challenges

Pathomics and DL are complementary approaches to computational image analysis with both operationally and research-driven applications. These approaches facilitate the discovery of new scientific knowledge and push the boundaries of clinical intuition. By contrast, operationally driven applications provide longer term, stable solutions to existing clinical use-cases, facilitate more efficient clinical workflows and enable the use of existing knowledge in new ways72. Research- and operationally driven applications are far from independent of one another, as an inherent feedback loop exists between them. Operationally driven applications designed on the basis of existing clinical needs often lead to unanticipated tertiary findings, which may motivate new research-driven applications, resulting in updated scientific knowledge to be eventually deployed in the form of new operationally driven applications. In the next decade, nephropathology will be indisputably altered by this feedback loop between AI-driven research discoveries and their application in clinical practice.

Research-driven applications

In contrast to oncology42,77,78,79,80,81,82,83,84,85, only a few examples exist of research-driven applications of computational image analysis in the nephrology literature72,106,107,108,109,110,111,112,113,114 (Table 3). The complexity of the renal parenchyma, the heterogeneity of the pathological features and the conventional use of multiple stains for renal histology preparations, make computer-aided image analysis of renal biopsy samples challenging (Table 4). In order to produce a complete map or atlas of the entire renal parenchyma and its pathological variations, subject matter experts must manually annotate a large number of normal and abnormal histological primitives (covering all normal structures and cells, and all possible pathological variations) to train the DL algorithms. Definition of the object boundaries (for example, glomerular tuft versus glomerular unit) is an obvious important pre-requisite. Although oncological pathologists can focus their manual annotation on a single monomorphic group of cells that form a specific tumour and then apply pathomic tools to identify subvisual variations, each form of kidney and glomerular disease is represented by a very heterogeneous aggregation and variable combination of normal and abnormal cells, even within the same disease category.

Table 4 Machine vision interrogation of nephropathology samples versus surgical pathology specimens
Full size table

The choice of methodology for each purpose is an additional challenge. For example, several methodologies have been proposed for the segmentation of glomeruli: supervised classification with a separate detection and segmentation stage, which is highly dependent on the precise annotation of the glomerulus; segmentation of the white sickle space surrounding the Bowman capsule, which is highly dependent on the glomerulus relative cutting plane; and domain adaptation methods, which are dependent on a number of annotated target and source domain training data (Table 3). Generalization of the selected approach is virtually impossible, as the number of manual annotations required for each histological primitive can also vary and is dependent on the intrinsic structural heterogeneity of the histological primitive. Furthermore, as each stain might provide unique information, the algorithms will need to be generated and tested on all stains before current practice can be changed to limit AI applications to a single stain. Finally, as the spatial distribution of different pathological structures is uniquely important to renal diagnostics, future applications in 3D WSI reconstruction115,116 will facilitate the accurate assessment of histological primitives (for example, glomerular number in a biopsy sample) and the spatial interplay between pathological features (for example, interstitial fibrosis and inflammation) and associated molecular signatures.

To date, the majority of studies using DL technology with kidney tissue have targeted low-level tasks, such as the detection and segmentation of normal histological primitives (for example, glomeruli, tubules, interstitial capillaries and arteries) and universally recognized pathological primitives (for example, segmental and global glomerulosclerosis, nodular glomerulosclerosis, interstitial fibrosis and tubular atrophy) on WSIs from paraffin72,73,111,117,118,119 and frozen sections120 (Table 3; Fig. 2). More complex tasks have been executed by using a DL architecture to stage diabetic glomerulosclerosis72 or to determine the association of interstitial fibrosis with clinical phenotypes such as chronic kidney disease stage, serum creatinine level, nephrotic range proteinuria at the time of the biopsy and renal survival at 1, 3 and 5 years (Fig. 2). Interestingly, the trained CNN models outperformed the pathologist-estimated fibrosis score in predicting the output classes106.

Many pathomic tools for the prediction of outcomes, genotype and disease mechanisms have not yet been applied to renal pathology. The current lack of training and knowledge of the principles of computer-aided image analysis and AI among pathologists for manual annotation and of data scientists for algorithm development are just two factors that have prevented this field from moving forwards faster.

The development of computational imaging tools for nephropathology and pathology can exemplify the concept of parallel discoveries (Fig. 2). For example, high-throughput gene expression analysis and comparison of multiple samples at the same time can be achieved by applying tissue microarrays to glass slides containing fragments of tissue from numerous patients121. This technique provides an opportunity to apply computational imaging techniques to capture the desired histological primitive from an image, match histological and cytological features across a range of length scales, and combine the information to create a single construct from multiple images in an array-based format (an image microarray), on multiple patients at the same time. The combination of tissue and image microarrays enables the assemblage of many fields, reflecting highly specific morpho-omic signatures122.

Similarly, the combination of laser capture microdissection techniques to isolate structures and cells can be enhanced via computational imaging techniques122. The pairing of computational image analysis with laser capture microdissection can be purposed to automatically detect the ROI as predefined by the pathologist, thus substituting the pathologist for the task, and/or to augment the process with information that would otherwise not be captured by the pathologist, but that can offer unique opportunities for omics discovery. For example, automatic detection of histological primitives followed by cluster analysis of those primitives may identify structures that share subvisual features independently of their morphological similarity at the human eye. Visualization tools to map these discoveries back to changes at the tissue level might improve our understanding of disease mechanisms123.

Arguably the most complex task currently underway in the field of nephropathology is the development of a functional kidney cell atlas by KPMP investigators. The development of a histology-based atlas of normal and pathological renal biopsy samples, where an array of validated morpho-genomic and morpho-transcriptomic data extractions will be spatially coupled to the WSI data, will provide an interactive computational framework in which interrogation can take place across all data domains, within an anatomic framework. This interactive framework is expected to provide deeper insights into cell and disease classification and lead to the identification of actionable targets at the molecular level. Unlike most prior efforts, the KPMP effort makes digital histology data a centre point for possible initial discovery modes.

Operationally driven applications

As research-driven applications become operationalized, AI tools for image analysis are likely to take a role in decision support through computer-assisted diagnosis (which, importantly, differs from computer-generated diagnosis). These decision support tools may be generated by processes such as DL algorithms, pathomics with feature engineering and computational integration of image data with data from other domains (Fig. 3). For example, automated detection of histological primitives has potential operational applications owing to its ability to increase the reproducibility of estimates of the distribution, severity and quantity of structures and lesions that are known predictors of outcome or molecular signatures but which have limited reproducibility when assessed manually (for example, globally sclerotic glomeruli, interstitial fibrosis, diabetic glomerulosclerosis stage)4,5,12,16,71; and to provide a quantitative estimate of histological primitives that can be visualized but cannot be reliably counted manually (for example, podocytes, interstitial capillaries)98,100,101,102,103,104. Thus, DL quantification of these histological primitives can augment the ability of the pathologist to interpret and diagnose kidney diseases.

Fig. 3: The nephropathology digital ecosystem.
figure 3

The digital ecosystem covers three phases: the digital pathology phase (analogue to digital conversion), the knowledge extraction phase, which relies on human and/or artificial intelligence (AI), and the actionable intelligence phase, in which integrated knowledge is applied to patient care. Each phase begins with input data and ends by generating output data that represent the input for the successive phase. In the digital pathology phase, glass slides from the renal biopsy sample (that is, the analogue input data), are converted into whole slide images (WSIs) (that is, digital output data). The WSIs represent the input data for the knowledge extraction phase, from which useful information is generated (output data). Knowledge extraction can be broken down into two types: human cognition and AI. AI techniques can be implemented as companion diagnostic tools alongside human cognition. Human cognition is employed when WSIs (input image data) are visually assessed or scored by a trained pathologist to generate diagnoses or morphological profiles as digital pathology-derived knowledge (output data). AI-based machine vision comprises both hand-crafted pathomics and discovery pathomics. For hand-crafted pathomics, image data (input) is transcribed into pathomic signatures (the pathomic feature space) and then translated into digital pathology-derived knowledge (the output data) using machine-learning models. For discovery pathomics, the image data (the input data) are transcribed or encoded into structured data and then decoded into an output signal (the output image data) using deep learning. Finally, in the actionable intelligence phase, the knowledge obtained from the digital images (input data) is integrated with other data types, for example, omics and clinical data, and used to diagnose, prognosticate and select targeted treatments (output data) for the patient.

Full size image

The information contained in WSIs that the human cognitive system cannot directly perceive, known as subvisual features, provide an opportunity to pose new scientific questions and novel insights that have not yet been conceived or imagined based on conventional pathology approaches. Although the implementation of these approaches and integration of new knowledge arising from these findings into clinical practice will be complex and require extensive validation, future ML pipelines will be an integral component of the pathologist’s workflow as decision support tools. Indeed, some of these algorithms are already making their way into routine practice in the field of radiation oncology42,124,125. The ultimate challenge will be the integration and implementation of computational image analysis-generated data with other knowledge domains to establish fused datasets and associated morpho-omic classifications, prognostications and predictions for kidney diseases at the individual patient level (that is, fused morpho-omic signatures). These applications will also have a role in drug discovery36, in the identification of participants for enrolment in clinical trials and in rationally determining the dosage of therapeutic agents.

Despite the excitement about these new approaches, pathologists and data scientists are advocating for cautious scepticism in the interpretation of quantitative results derived from the application of image analysis techniques115,116,117,118,119. Similar caution is warranted when considering the potential clinical implementation of such approaches, recognizing that vigorous validation should always be an integral step of the deployment process126,127,128,129,130. Our understanding of the key drivers behind AI-based determinations is still incomplete, and we still do not possess sufficiently developed tools to assess performance metrics or enforce QC measures in the clinical setting. As algorithms become available for clinical deployment, regulatory bodies will need to work closely with the other stake-holders (such as pathologists, computer scientists, nephrologists, pathology laboratories and institutions) to establish a framework for their approval and implementation.

Limitations of sample size

As digital pathology becomes more widely accessible, the analytical tools of AI and ML will reveal new opportunities with which to develop large-scale, kidney-centric data ecosystems, to combine datatypes and datasets for discovery and validation, and for the practice of precision nephrology (Table 2). However, ML models are based on learning patterns in data and require sufficient numbers of training examples, with the added observation that most AI-driven solutions require large amounts of data during their training process to be both successful and generalizable. This requirement could prove to be a challenge in their application to rare kidney diseases, where sample sizes are inherently small. Fortunately, novel AI and ML approaches that can be applied to small training sets are now becoming available. These approaches use methodologies such as data augmentation, whereby computational perturbations (usually of a spatial or colorimetric nature) are used to increase the variation of a training set to improve model generalization; new approaches to high-dimensional data clustering131, whereby samples are grouped into similar categories on the basis of intrinsic data properties without the need for labelled training examples; or unsupervised machine-learning processes, which provide reasonably robust image segmentation performance when used to identify normal renal histological primitives100,118. All these capabilities, taken together, represent a promising future for these new computational tools, and we expect that they will augment the clinical practice of both nephrology and pathology in the near future.

The limitations of ML approaches for use on small datasets will also be addressed by the development of robust learning health-care systems. Aided by AI technology, learning health-care systems rely on the data-driven generation of knowledge to enable the practice of evidence-based medicine132,133,134,135,136. Implementation of this technology-driven health-care model will facilitate the sharing of data and ML hyper-parameters (that is, parameters that refer to the model selection task or to the quality and speed of the learning-process algorithm) between institutions, leading to aggregate models with better potential for generalizability. Furthermore, the continued establishment and management of large international nephrology consortia will also have a key role in the success of AI applications in rare kidney diseases. Given the nature of ongoing consortium-driven kidney research, which is most typically based on cross-institutional and multi-omics data13,137, it is highly likely that these consortia will serve as the blueprint for future collaborative AI endeavours in the renal space that operate at scale.

Training of medical professionals

Renal pathologists will have a critical role in establishing the digital nephropathology ecosystem, developing clinically relevant ML and AI algorithms, and deploying such algorithms in routine clinical practice. Renal pathologists are the subject matter experts in renal histology and, as such, they will need to work closely with data and machine vision scientists to design studies and protocols, ensure that relevant WSI datasets are made available, annotate WSI libraries, and finally curate, validate and interpret the data that arise from these efforts. It will be critical for pathologists to be well-versed in AI and ML applications and methods, including their limitations, so that these powerful tools can be applied to clinical practice quickly, effectively and reliably. Owing to the interdisciplinary nature of integrated computational pathology, the need for nephrologists to acquire new knowledge and skills is equally germane, as these individuals represent the penultimate consumers of this new class of information as generated by these approaches. Acknowledgment that AI will be integral part of medicine is the first step towards the re-design of curricula for medical schools and residency or fellowship training programmes to address the current inadequacy in preparing medical professionals for the imminent future138 (Table 2).

Conclusions

We can borrow concepts and analogies from biology to summarize the transformative process that is beginning to affect the nephropathology community. This transformative process is based on a variety of events that represent the first decisive steps in the long anticipated pathway towards a new digitally enhanced ecosystem139. Humans are learning to operate within the confines of this new paradigm, as represented by digital avatars of living components, the operability of which is closely linked to the non-living components of their environment. These biotic and abiotic constituents are linked together by new, interdependent cycles, dynamisms and flows. The next-generation nephropathology is expected to operate within this new digital ecosystem and apply artificial (augmenting) intelligence tools to implement human–machine synergistic protocols in clinical practice to better categorize, predict and prognosticate kidney diseases, and to identify candidates for conventional or novel therapeutic approaches. The critical steps needed for the success of this transformational process ultimately depend on practitioners in the field being open to acquiring new skills, and the adaptation of practice patterns to support the new discovery environments made possible by these technologies. Most importantly, it will be essential for renal pathologists and nephrologists to understand both the strengths and inherent limitations of digital pathology technologies and, at the same time, be willing to invest the time and effort needed to shoulder their adoption.