PT  - JOURNAL ARTICLE
AU  - T. Campbell Arnold
AU  - Steven N. Baldassano
AU  - Brian Litt
AU  - Joel M. Stein
TI  - Simulated Diagnostic Performance of Ultra-Low-Field MRI: Harnessing Open-Access Datasets to Evaluate Novel Devices
AID  - 10.1101/2021.07.02.21259789
DP  - 2021 Jan 01
TA  - medRxiv
PG  - 2021.07.02.21259789
4099  - http://medrxiv.org/content/early/2021/07/05/2021.07.02.21259789.short
4100  - http://medrxiv.org/content/early/2021/07/05/2021.07.02.21259789.full
AB  - The purpose of this study is to demonstrate a method for virtually evaluating novel imaging devices using machine learning and open-access datasets, here applied to a new, ultra-low-field strength (ULF), 64mT, portable MRI device. Paired 3T and 64mT brain images were used to develop and validate a transformation converting standard clinical images to ULF-quality images. Separately, 3T images were aggregated from open-source databases spanning four neuropathologies: low-grade glioma (LGG, N=76), high-grade glioma (HGG, N=259), stroke (N=28), and multiple sclerosis (MS, N=20). The transformation method was then applied to the open-source data to generate simulated ULF images for each pathology. Convolutional neural networks (DenseNet-121) were trained to detect pathology in axial slices from either 3T or simulated 64 mT images, and their relative performance was compared to characterize the potential diagnostic capabilities of ULF imaging. Algorithm performance was measured using area under the receiver operating characteristic curve. Across all cohorts, pathology detection was similar between 3T and simulated 64mT images (LGG: 0.97 vs. 0.98; HGG: 0.96 vs. 0.95; stroke: 0.94 vs. 0.94; MS: 0.90 vs 0.87). Pathology detection was further characterized as a function of lesion size, intensity, and contrast. Simulated images showed decreasing sensitivity for lesions smaller than 4 cm2 (∼2.25 cm in diameter). While simulations cannot replace prospective trials during the evaluation of medical devices, they can provide guidance and justification for prospective studies. Simulated data derived from open-source imaging databases may facilitate testing and validation of new imaging devices.HighlightsUltra-low-field, point-of-care MRI has potential to detect a range of pathologies including brain tumors, strokes, and multiple sclerosis. However, determining the diagnostic capabilities and appropriate use case for such devices requires further prospective studies.Open-source image datasets provide a powerful tool for accelerating imaging research and enable simulated trials that can guide prospective clinical trials or device development.Competing Interest StatementJoel Stein is the Principal Investigator on a Sponsored Research Agreement to test the Hyperfine ULF MRI device in an ambulatory setting for imaging hydrocephalus. Dr. Stein does not personally receive any compensation from this company. Brian Litt is an unpaid Scientific Advisor to 4Catalyzer, a Jonathan Rothberg founded incubator, which initiated Hyperfine as one of its companies. Dr. Litt is a Co-Founder of EpilepsyCo, a separate Jonathan Rothberg founded company. His time consulting for EpilepsyCo is compensated under a consulting agreement. Both Dr. Stein's and Dr. Litt's interactions with these companies, and those of their trainees, are performed in strict accordance with the policies and conflict of interest management rules of the University of Pennsylvania and are reviewed annually.Funding StatementThis work was supported by a sponsored research agreement with Hyperfine. Additional funding was provided by the NIH (T32NS091006-01), the HHMI-NIBIB Interfaces Initiative (5T32EB009384-10), Jonathan and Bonnie Rothberg, The Mirowski Family Fund, and Neil and Barbara Smit.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Data was collected as part of an ongoing research study approved by the University of Pennsylvania Institutional Review Board and patients provided informed consent.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesAll code related to simulated image generation, classifier design, and statistical analysis can be found at: https://github.com/tcama/SCT and https://github.com/penn-cnt/Arnold_simulated_clinical_trial. https://www.oasis-brains.org/ http://www.ia.unc.edu/MSseg/ https://www.med.upenn.edu/cbica/brats2019.html http://www.isles-challenge.org/ISLES2015/ https://portal.fli-iam.irisa.fr/english-msseg/ (ULF)Ultra-low-field strength(HF)high-field strength(LGG)low-grade glioma(HGG)high-grade glioma(MS)multiple sclerosis(FLAIR)fluid-attenuated inversion recovery(SNR)signal to noise ratio(AUC)area under curve(ROC)receiver operating characteristic(CAMs)class activation maps(EFS)early feasibility studies(IDE)investigational device exemption