Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams

View ORCID ProfileYiqiu Shen, Farah E. Shamout, Jamie R. Oliver, Jan Witowski, Kawshik Kannan, Jungkyu Park, Nan Wu, Connor Huddleston, Stacey Wolfson, Alexandra Millet, Robin Ehrenpreis, Divya Awal, Cathy Tyma, Naziya Samreen, Yiming Gao, Chloe Chhor, Stacey Gandhi, Cindy Lee, Sheila Kumari-Subaiya, Cindy Leonard, Reyhan Mohammed, Christopher Moczulski, Jaime Altabet, James Babb, Alana Lewin, Beatriu Reig, Linda Moy, Laura Heacock, Krzysztof J. Geras
doi: https://doi.org/10.1101/2021.04.28.21256203
Yiqiu Shen
1Center for Data Science, New York University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Yiqiu Shen
Farah E. Shamout
2Engineering Division, NYU Abu Dhabi
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jamie R. Oliver
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jan Witowski
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kawshik Kannan
4Department of Computer Science, Courant Institute, New York University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jungkyu Park
5Vilcek Institute of Graduate Biomedical Sciences, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nan Wu
1Center for Data Science, New York University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Connor Huddleston
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Stacey Wolfson
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alexandra Millet
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Robin Ehrenpreis
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Divya Awal
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Cathy Tyma
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Naziya Samreen
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yiming Gao
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Chloe Chhor
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Stacey Gandhi
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Cindy Lee
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sheila Kumari-Subaiya
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Cindy Leonard
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Reyhan Mohammed
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christopher Moczulski
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jaime Altabet
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
James Babb
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alana Lewin
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Beatriu Reig
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Linda Moy
3Department of Radiology, NYU Grossman School of Medicine
5Vilcek Institute of Graduate Biomedical Sciences, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Laura Heacock
3Department of Radiology, NYU Grossman School of Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Krzysztof J. Geras
3Department of Radiology, NYU Grossman School of Medicine
5Vilcek Institute of Graduate Biomedical Sciences, NYU Grossman School of Medicine
1Center for Data Science, New York University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: k.j.geras@nyu.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

Ultrasound is an important imaging modality for the detection and characterization of breast cancer. Though consistently shown to detect mammographically occult cancers, especially in women with dense breasts, breast ultrasound has been noted to have high false-positive rates. In this work, we present an artificial intelligence (AI) system that achieves radiologist-level accuracy in identifying breast cancer in ultrasound images. To develop and validate this system, we curated a dataset consisting of 288,767 ultrasound exams from 143,203 patients examined at NYU Langone Health, between 2012 and 2019. On a test set consisting of 44,755 exams, the AI system achieved an area under the receiver operating characteristic curve (AUROC) of 0.976. In a reader study, the AI system achieved a higher AUROC than the average of ten board-certified breast radiologists (AUROC: 0.962 AI, 0.924±0.02 radiologists). With the help of the AI, radiologists decreased their false positive rates by 37.4% and reduced the number of requested biopsies by 27.8%, while maintaining the same level of sensitivity. To confirm its generalizability, we evaluated our system on an independent external test dataset where it achieved an AUROC of 0.911. This highlights the potential of AI in improving the accuracy, consistency, and efficiency of breast ultrasound diagnosis worldwide.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This work was supported in part by grants from the National Institutes of Health (P41EB017183, R21CA225175), the National Science Foundation (1922658), the Gordon and Betty Moore Foundation (9683), the Polish National Agency for Academic Exchange (PPN/IWA/2019/1/00114/U/00001) and NYU Abu Dhabi.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This retrospective study was approved by the NYU Langone Health Institutional Review Board (ID#i18-00712_CR3) and is compliant with the Health Insurance Portability and Accountability Act. Informed consent was waived since the study presents no more than minimal risk.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

The external test dataset is publicly available at https://scholar.cu.edu.eg/?q=afahmy/pages/dataset. The NYU Breast Ultrasound Dataset is not currently permitted for public release by the institutional review board. We published the following report explaining how the dataset was created for reproducibility: https://cs.nyu.edu/~kgeras/reports/ultrasound_datav1.0.pdf.

https://scholar.cu.edu.eg/?q=afahmy/pages/dataset

https://cs.nyu.edu/~kgeras/reports/ultrasound_datav1.0.pdf

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted April 30, 2021.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams
Yiqiu Shen, Farah E. Shamout, Jamie R. Oliver, Jan Witowski, Kawshik Kannan, Jungkyu Park, Nan Wu, Connor Huddleston, Stacey Wolfson, Alexandra Millet, Robin Ehrenpreis, Divya Awal, Cathy Tyma, Naziya Samreen, Yiming Gao, Chloe Chhor, Stacey Gandhi, Cindy Lee, Sheila Kumari-Subaiya, Cindy Leonard, Reyhan Mohammed, Christopher Moczulski, Jaime Altabet, James Babb, Alana Lewin, Beatriu Reig, Linda Moy, Laura Heacock, Krzysztof J. Geras
medRxiv 2021.04.28.21256203; doi: https://doi.org/10.1101/2021.04.28.21256203
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams
Yiqiu Shen, Farah E. Shamout, Jamie R. Oliver, Jan Witowski, Kawshik Kannan, Jungkyu Park, Nan Wu, Connor Huddleston, Stacey Wolfson, Alexandra Millet, Robin Ehrenpreis, Divya Awal, Cathy Tyma, Naziya Samreen, Yiming Gao, Chloe Chhor, Stacey Gandhi, Cindy Lee, Sheila Kumari-Subaiya, Cindy Leonard, Reyhan Mohammed, Christopher Moczulski, Jaime Altabet, James Babb, Alana Lewin, Beatriu Reig, Linda Moy, Laura Heacock, Krzysztof J. Geras
medRxiv 2021.04.28.21256203; doi: https://doi.org/10.1101/2021.04.28.21256203

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Radiology and Imaging
Subject Areas
All Articles
  • Addiction Medicine (157)
  • Allergy and Immunology (412)
  • Anesthesia (90)
  • Cardiovascular Medicine (852)
  • Dentistry and Oral Medicine (156)
  • Dermatology (97)
  • Emergency Medicine (247)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (392)
  • Epidemiology (8527)
  • Forensic Medicine (4)
  • Gastroenterology (381)
  • Genetic and Genomic Medicine (1731)
  • Geriatric Medicine (166)
  • Health Economics (370)
  • Health Informatics (1232)
  • Health Policy (615)
  • Health Systems and Quality Improvement (465)
  • Hematology (196)
  • HIV/AIDS (367)
  • Infectious Diseases (except HIV/AIDS) (10258)
  • Intensive Care and Critical Care Medicine (550)
  • Medical Education (192)
  • Medical Ethics (51)
  • Nephrology (209)
  • Neurology (1662)
  • Nursing (97)
  • Nutrition (247)
  • Obstetrics and Gynecology (321)
  • Occupational and Environmental Health (450)
  • Oncology (918)
  • Ophthalmology (261)
  • Orthopedics (99)
  • Otolaryngology (172)
  • Pain Medicine (110)
  • Palliative Medicine (40)
  • Pathology (249)
  • Pediatrics (532)
  • Pharmacology and Therapeutics (245)
  • Primary Care Research (205)
  • Psychiatry and Clinical Psychology (1751)
  • Public and Global Health (3814)
  • Radiology and Imaging (620)
  • Rehabilitation Medicine and Physical Therapy (316)
  • Respiratory Medicine (518)
  • Rheumatology (204)
  • Sexual and Reproductive Health (164)
  • Sports Medicine (156)
  • Surgery (188)
  • Toxicology (36)
  • Transplantation (99)
  • Urology (74)