ABSTRACT
Background The diagnosis of cancer in thyroid nodules with indeterminate cytology (Bethesda III/IV) is challenging as fine-needle aspiration (FNA), the gold standard method, has limitations, and these cases usually require diagnostic surgery. As approximately 77% of these nodules are not malignant, a diagnostic test accurately identifying benign thyroid nodules can reduce surgery rates. We have previously reported the development and validation of a microRNA-based thyroid molecular classifier for precision endocrinology (mir-THYpe) with high sensitivity and specificity, which could be performed directly from readily available cytological smear slides without the need for a new dedicated FNA. We sought to evaluate whether the use of this test in real-world clinical routine can reduce the rates of surgeries for Bethesda III/IV thyroid nodules and analyze the test performance.
Methods We designed a real-world, prospective, multicenter cohort study. Molecular tests were performed in a real-world clinical routine with samples (FNA smear slides) prepared at 128 cytopathology laboratories. Patients were followed-up from March 2018 until surgery or until March 2020 (for those patients not recommended for surgery). The final diagnosis of thyroid tissue samples was retrieved from postsurgical anatomopathological reports.
Results After applying the exclusion criteria, 435 patients (440 nodules) classified as Bethesda III/IV were followed-up. The rate of avoided surgeries was 52.5% for all surgeries and 74.6% for “potentially unnecessary” surgeries. After the statistical treatment of non-resected test-negative samples, the test achieved 89.3% sensitivity (95% CI 82–94.3), 81.65% specificity (95% CI 76.6–86), 66.2% positive predictive value (95% CI 60.3–71.7), and 95% negative predictive value (95% CI 91.7–97) at 28.7% (95% CI 24.3–33.5) cancer prevalence. The test influenced 92.3% of clinical decisions.
Conclusions The reported data demonstrate that the use of the microRNA-based classifier in the real-world can reduce the rate of thyroid surgery with robust performance and significantly influence clinical decision-making.
INTRODUCTION
Thyroid cancer has increased in the last few decades, and it ranks as the ninth most-incident cancer worldwide.1 Although there is consensus that overdiagnosis is the main reason for increased incidence 2,3 mainly due to incidental thyroid nodules found on imaging screenings performed for reasons other than thyroid disease evaluation,4 there has also been a true increase in the occurrence of thyroid cancer.5
Fine-needle aspiration (FNA) cytology is the current gold standard for triaging patients with suspicious thyroid nodules detected clinically or on ultrasound (US),6-8 and the six-tier Bethesda System for Reporting Thyroid Cytopathology attempts to standardize cytopathologic analysis.9
Approximately 64% of histologically diagnosed thyroid cancer nodules were initially classified on FNA cytology as Bethesda V or VI, with a combined risk of malignancy (RoM) of 91% (only 1.1 surgeries required to detect one case of cancer). Although the proportion of histologically diagnosed thyroid cancer nodules initially classified on FNA cytology as “indeterminate” (Bethesda III and IV) is lower (∼29%), the combined RoM for these two classes is about 23% only,10 implying that around 77% of surgeries are potentially unnecessary and could have been reconsidered or avoided (4.4 surgeries required to detect one case of cancer).
Molecular testing has emerged as an option to reduce the need for diagnostic surgery in indeterminate thyroid nodules,8 recommended by the American Thyroid Association (ATA)11 and National Comprehensive Cancer Network (NCCN)12 guidelines. We have recently reported the development and validation of a new microRNA-based thyroid molecular classifier test for a precision endocrinology (mir-THYpe) with high sensitivity and specificity, which could be performed directly from readily available cytological smear slides at a significantly lower cost without the need for a new dedicated FNA.13,14
To evaluate the impact of mir-THYpe results on reducing the rates for surgery in the real-world clinical settings and analyze its performance on Bethesda III and IV nodules, we present data of a real-world, prospective, and multicenter study.
METHODS
Study Population and Design
As we conducted a real-world evidence study, there was no specific method of patient recruitment. Patients eligible for this study had a signed medical prescription for the mir-THYpe test in a real-world clinical routine with at least one thyroid nodule previously biopsied by FNA and classified as Bethesda III or IV. Patients were 18 years or older, provided written informed consent, and paid out-of-pocket for the test. The study was approved by the board of the investigational ethics committee.
The mir-THYpe molecular test was performed, and the report was released. Regardless of the results (positive or negative for malignancy), patients were assigned to one of the two groups according to the physician’s recommendation: Group A underwent a thyroidectomy (tests performed March 2018–March 2020); Group B avoided thyroid surgery (tests performed March 2018–October 2019, follow-up in March 2020). Patients who initially received no recommendations for surgery but later underwent thyroidectomy (regardless of the reason) were included in Group A.
Patients who underwent thyroid surgery but for whom we could not obtain original postsurgical anatomopathological (AP) reports were excluded. Samples in which the FNA cytology report and other characteristics, such as size and location, did not correlate with the nodule described in the AP report were also excluded from the performance analysis.
Samples
The FNA cytology Bethesda classes and the postsurgical histological classification of resected nodules were not revised by other pathologists. The FNA smear slides of the 440 samples (used for the mir-THYpe test – Figure 1, box c) were prepared, and the Bethesda classes were assigned, by 128 cytopathology laboratories across 54 cities in all five regions and the Federal District of Brazil (Figure S1-Supplementary Material). All fixation and staining protocols of the FNA cytology smear slides were accepted. The stored samples were sent without temperature control (room-temperature). The 168 postsurgical AP reports with the histological classification of nodules (Figure 1, boxes h and o) were assigned by 53 pathology laboratories across 21 cities in three regions of Brazil.
Although the indolent behavior of the noninvasive follicular thyroid neoplasms with papillary-like nuclear features (NIFTPs), its diagnosis is established only after applying stringent histological criteria.15 As NIFTPs require surgical treatment, positive mir-THYpe tests were considered true-positives. Data on nodule size were retrieved from postsurgical AP reports.
Molecular Analysis
The mir-THYpe molecular classifier test was performed according to the previously described method.13 In brief, thyroid cells from the FNA smear slides were microdissected and submitted to manual RNA extraction, followed by reverse-transcription cDNA synthesis, preamplification stage, and real-time qPCR. The normalized Ct values for the expression of 11 microRNAs were submitted to a proprietary tree-based model that classified samples as “negative” or “positive” for malignancy. A 12th microRNA (miR-375) was also analyzed to screen for medullary thyroid cancer.16 No mutation or DNA analyses were performed on the samples.
Statistical Analyses
Statistical analyses were performed using R software, an open-source statistical programming environment. The confidence intervals for sensitivity, specificity, and accuracy were the “exact” Clopper– Pearson confidence intervals, and the confidence intervals for the predictive values were the standard logit confidence intervals.17 Confidence intervals for the means were calculated using the method described by Walline.18
RESULTS
Patient and Nodule Characteristics
A total of 481 real-world patients with 488 nodules (1.01 nodule/patient) and a medical prescription for the mir-THYpe test provided written informed consent. Of these, 39 nodules (8%) were excluded due to loss to follow-up. Eight nodules (1.6%) classified as Bethesda V and one nodule (0.2%) classified as Bethesda II were also excluded (Figure 1, box b). The remaining 440 nodules (from 435 patients; 1.01 nodules/patient) classified as Bethesda III and IV, fulfilled all initial inclusion criteria and were categorized according to the mir-THYpe test results (Figure 1 and Table 1).
Although the mir-THYpe test was negative for malignancy in 239 samples (Benign Call Rate [BCR] of 54.3%), 17 nodules (7.1%) underwent thyroidectomy (Group A), (9 previously classified as Bethesda III [52.9%] and 8 as Bethesda IV [47.1%]). The mean age of these 17 patients was 49 years, the mean nodule size was 1.8 cm, and the average test-to-surgery time was 170 days; for details, see Table S1 in the Supplementary Material. The mean age of the other 222 patients (92.9%) (Group B) was 52 years. They received no medical recommendation for surgery after the mir-THYpe test result (as of March 2020); the mean follow-up time was 412.9 days. Of these, 117 nodules (52.7%) were classified as Bethesda III and 105 (47.3%) as Bethesda IV; Table S2 in the Supplementary Material presents the 95% confidence intervals and ranges. Test-to-surgery and follow-up times are described in Table S3 in the Supplementary Material.
The mir-THYpe test was positive for malignancy in 201 samples (Positive Call Rate [PCR] of 45.7%), and 184 nodules (91.6%) were surgically treated (Group A). However, 25 nodules (13.6%) were excluded due to lack of access to the original postsurgical AP report, and 8 samples (4.3%) were excluded because the punctured nodule described in the FNA cytology did not correlate with the nodule(s) described in the postsurgical AP report. Of the remaining 151 resected nodules (82.1%), 83 were previously classified as Bethesda III (55%) and 68 as Bethesda IV (45%). The mean age of the patients was 49 years, the mean nodule size was 1.13 cm, and the average test-to-surgery time was 79 days. The average age of the other 17 patients (8.4%) (Group B) was 54 years, and the patients were not surgically treated (as of March 2020), despite a positive result; the mean follow-up time was 359.6 days. Among them, 13 nodules (76.5%) were classified as Bethesda III and 4 (23.5%) as Bethesda IV; Table S2 in the Supplementary Material presents the 95% confidence intervals and ranges. Test-to-surgery and follow-up times are described with periods in Table S3 in the Supplementary Material.
Overall, the “typical” profile of a real-world patient referred for the mir-THYpe test was female (82%), younger than 55 years (57.3%) with a Bethesda III (56.1%) thyroid nodule <2 cm (86.3%). Table 1 displays the demographic and clinical characteristics of the study cohort.
Test Performance
Table 2 summarizes the test performance across all 168 resected nodules according to the histopathological type found post-surgery.
All 17 samples with negative results treated with surgery, regardless of Bethesda class, were confirmed as benign lesions (true-negatives) in the postsurgical AP reports, achieving an overall Negative Predictive Value (NPV) of 100% (one-side 97.5% CI 80.5–100). This result must be interpreted with caution because the sample size was too small (patients with a negative test usually do not receive a surgery recommendation). This result included four Hürthle cell adenomas (23.5% [95% CI 6.8–49.9]).
Among the 151 resected nodules with positive results included in the final analysis, 100 nodules were confirmed as cancer/NIFTP lesions (true-positives) in the postsurgical AP report, including nine NIFTPs (9% [95% CI 4.2–16.4]) and one medullary carcinoma (1% [95% CI 0.02–5.5], achieving an overall Positive Predictive Value (PPV) of 66.2% (95% CI 60.3–71.7). Splitting the results by cytological class when the mir-THYpe test was positive (Table 3), 52 of the 83 Bethesda III (62.7%), and 48 of the 68 Bethesda IV nodules (70.6%) were accurately classified. Fifty-one nodules showed false-positive results due to benign tissue identified in the postsurgical AP reports (33.8% [95% CI 26.3–41.9]).
Overall, the test accuracy was 70% (95% CI 62–76.5) as it accurately classified 117 of the 168 resected samples, regardless of test results.
Although expected, the small number of resected test-negative nodules impaired the realistic calculation of the sensitivity and NPV (both at 100%) and increased the disease prevalence to unrealistic values (60%). To minimize this effect, we performed a theoretical calculation applying the sensitivity observed during the validation study13 (94.6%) to the 222 non-resected nodules with test-negative results (Figure 1, box f), resulting in 210 true-negatives and 12 false-negatives. In this scenario, the mir-THYpe test achieved 89.3% sensitivity (95% CI 82–94.3), 81.65% specificity (95% CI 76.6–86), 66.2% positive predictive value (PPV) (95% CI 60.3–71.7), and 95% negative predictive value (NPV) (95% CI 91.7–97) at a 28.7% (95% CI 24.3– 33.5) disease prevalence (Table 4).
Comparison with other Thyroid Molecular Classifier Tests
To evaluate the real-world utility of the mir-THYpe test, we performed a comparison with the results of prospective studies on the latest version of two leading commercially available thyroid molecular classifier tests: ThyroSeq v319 and Afirma GSC.20 We also compared the molecular test results with the performance of the FNA alone using data from a meta-analysis10 (Table 3).
The true-positive rate of the mir-THYpe test (66.2%) for Bethesda III and IV cytology was very similar to that in the ThyroSeq v3 prospective study (66%), but the number of tested samples in the present study was 35% higher (151 vs. 97). The mir-THYpe showed significantly better performance than the results for Afirma GSC (45.8%), and all molecular tests showed better results than those obtained with FNA cytology alone (22.6%). It is noteworthy that among the compared molecular classifiers, ours is the only study to perform the test exclusively with FNA smear slides and a real-world design.
Influence on Clinical Decisions and Surgery Reduction Rates
The mir-THYpe test influenced 93% of clinical decisions when the result was negative for malignancy (222 nodules with no medical recommendation for surgery, out of 239 test-negatives). The positive test results influenced 91.5% of clinical decisions (184 nodules treated surgically, out of 201 test-positives). Overall, the mir-THYpe test influenced 92.3% of clinical decisions (406 out of 440 cases).
To calculate the rates of total avoided surgeries among the 440 eligible patients, we assumed that 423 would have undergone thyroidectomy if no molecular test was available (we subtracted 17 patients not treated surgically, despite a positive result — Figure 1, box k). Of these, 222 patients avoided surgery (Figure 1, box f), resulting in 52.5% (95% CI 47.6–57.3) of all surgeries.
Following Figure 1, we considered that 280 patients (17 [box e] + 222 [box f] + 51 [box o]) had a benign nodule and would undergo a potentially unnecessary surgery if no molecular test was available. As we could not assume that all 222 samples from box f were truly benign, we applied the sensitivity observed in the validation study13 (94.6%) and estimated that around 210 samples were true-negative nodules in this cohort, resulting in a total of 268 patients who would have underwent a potentially unnecessary surgery if no molecular test was available. Considering that only 68 patients (17 [box e] + 51 [box o]) were surgically treated, the mir-THYpe test avoided 74.6% (95% CI 69–79.2) of potentially unnecessary surgeries.
DISCUSSION
The present study aims to prospectively evaluate the real-world performance, impact on clinical decisions, and the true potential of a previously described microRNA-based thyroid molecular classifier for avoiding surgeries in a multicenter cohort.13
Although randomized controlled trials (RCTs) are considered the gold standard for analyzing the efficacy of therapies, they are limited to a subset of patients who are not fully representative of the unselected real-world patients.21-23 Real-world evidence studies better represent routine practice when compared with idealized conditions of RCTs, thus providing a valuable reflection of the range and distribution of patients observed in clinical practice.22-24 A real-world evidence study design can help physicians understand the practical use of molecular tests, providing a clear view of what to expect.
One main advantage was that the submitted specimens belonged to patients with a real clinical indication for testing and/or a desire to avoid surgery, thus avoiding the bias of including samples based on the Bethesda classification only. Moreover, bias was minimized as the patients paid out-of-pocket for the tests. Another advantage was that we could not control where the samples came from; FNA smear slides were sent at room-temperature from anywhere, and there were no boundaries to guarantee a highly heterogeneous cohort. The FNA smear slides were prepared at 128 cytopathology laboratories, ensuring a highly heterogeneous sample cohort and robustness to analyze the samples (prepared using different fixation and staining protocols by various operators). As summarized in Table 1, patient demographics and nodule characteristics were in accordance with expected real-world proportions, illustrated in the “typical” patient profile described above. Lastly, the patient and physician received the molecular test results before deciding the next step (unlike blinded studies in which all nodules are resected, and test influence is unmeasurable). Therefore, we could follow the patients and observe the real impact of their more-informed clinical decisions and test utility on surgery avoidance. The high impact of the mir-THYpe test on real-world clinical decision-making (92.3% overall) re-emphasizes that the Bethesda categories III and IV present a challenge for physicians to decide the best option for their patients. 74.5% of potentially unnecessary surgeries were avoided, suggesting that the molecular test could minimize this problem and potentially improve the efficiency of the healthcare system.
However, our study had some limitations. The multicenter design made it difficult to closely monitor patients as they were not centralized to a few institutions, and there was limited access to other clinical parameters. Ideally, non-operated patients should be followed with US at least to identify possible evolutions in the nodules suggestive of management changes. Although the physicians routinely monitored the patients, we had restricted access. Therefore, we had to limit the follow-up monitoring according to whether the patient underwent thyroid surgery or not. Moreover, we did not review the patients’ FNA or AP slides in order to guarantee the real-world approach, even though we assumed the risk of potential misclassifications at the pathology laboratories. Finally, as the majority of patients who tested negative were not submitted to thyroid surgery (222/239, 93%), we did not have the statistical power to evaluate the performance of negative results (mainly the sensitivity and the NPV). Although all 17 resected nodules with negative results were confirmed benign, the sample size impaired the confidence on 100% sensitivity and NPV. However, the statistical adjustment applied (described above on Results) for the 222 test-negative samples to minimize this effect and calculate the test performance was valid as the sensitivity of a diagnostic test is a parameter not influenced by disease prevalence. Thus, on this scenario, the present study achieved an NPV of 95%, which seems realistic (similar to the NPV of 95.9% observed in the previous validation study13).
The statistical parameters analyzing the performance of a test are crucial in determining practical applications and adoption. ThyroSeq v3 and Afirma GSC are the two main available thyroid molecular classifiers, widely adopted worldwide with proven clinical impact. Hence, we compared the mir-THYpe test performance observed in this study with prospective studies describing the performance of the latest version of these two tests.19,20 Based on the true-positives for each test, the mir-THYpe achieved similar performance to the ThyroSeq v3 test (62.7% vs. 64%, 70.6% vs. 68.1%, and 66.2% vs. 66% for Bethesda III, IV, and overall, respectively) and better performance than the Afirma GSC in all scenarios (62.7% vs. 44.7%, 70.6% vs. 47.4%, and 66.2% vs. 45.8% for Bethesda III, IV, and overall) (See Table 3 for details and 95% confidence intervals). In practical and quantitative terms, when either mir-THYpe or ThyroSeq v3 is used alongside FNA indeterminate cytology, around 1.5 surgeries are required to identify 1 cancer patient. If Afirma GSC is chosen, 2.18 surgeries are required to identify 1 cancer patient (1.45 more than when using mir-THYpe) and 4.43 surgeries are required with FNA alone (2.94 more than adding mir-THYpe) (See Table S4 in the Supplementary Material). Based on the NPV of each test, the mir-THYpe achieved similar performance (95%) to ThyroSeq v3 (97%) and Afirma GSC (96%). Moreover, the sample type requested to perform the test is also relevant, because if no material was stored from the first FNA (frozen), ThyroSeq v3 and Afirma GSC need a new and dedicated FNA. Therefore, patients need to attend a restricted collection site and undergo another invasive procedure. Furthermore, delicate logistics operations are needed, as the samples must be refrigerated and delivered on time. For physicians, there is no guarantee that the new puncture will have the same cellular composition as the previous sample that indicated the need for molecular testing or that the logistics chain will maintain the sample at the appropriate temperature and ensure timely delivery. Although ThyroSeq v3 is under validation for FNA smear slides, the available results are limited to only 31 samples with different performance depending on slide staining (Papanicolau 100%, Diff-Quick 65%, and overall 81%).25 Furthermore, FNA smear slides were not used for the prospective study validation.19
In summary, the reported data demonstrate over a heterogeneous and large cohort of patients that the real-world use of the mir-THYpe test can reduce the rates of surgeries for Bethesda III and IV indeterminate cytology nodules and significantly influence clinical decisions. Moreover, the real-world test performance is robust and comparable to other well-established thyroid molecular classifiers. Furthermore, it can be performed using already available cytology smear slides at a significantly lower cost. Cost-effectiveness analyses of the test’s financial impact and healthcare system savings are under development, and together with the present data, will help to guide new policies.
Data Availability
Due to the nature of this research, to protect participants' privacy and confidentiality, raw data are not shared publicly. Relevant data supporting the findings and conclusions of this study are available within the article and its supplementary materials. Upon a justifiable request, the share of de-identified data should be approved by the board of an investigational ethics committee.
Author Contributions
MTS designed and supervised the study. DLAF, GMM, RAS, CF, GFR, RC, HER, FV and MV contributed with data and medical expertise. BMR and SS contributed with data collection, integration, and patients’ follow-up. MTS drafted the manuscript. MTS and BMR contributed as molecular biologists. All authors read and approved the final version.
Conflict of interests
MTS holds equity at ONKOS Molecular Diagnostics. BMR and SS are formal employees at ONKOS Molecular Diagnostics. The other authors declare no competing interests.
Funding
Onkos Molecular Diagnostics and FAPESP – Fundação de Amparo à Pesquisa do Estado de São Paulo (Project 2017/16417-9)
SUPPLEMENTARY MATERIAL
Geographic distribution of the 128 cytopathology labs where the 440 FNA smear slides tested in the study were prepared and the Bethesda categories assigned.
Clinical details of the 17 patients with a test-negative result with nodules surgically resected (all true-negatives).
Calculated number of surgeries needed to identify 1 case of cancer through molecular tests and FNA alone, according to the results of each study.
Acknowledgments
We are grateful to Andrei Felix for laboratory support, and the SUPERA Technology and Innovation Park team for institutional support. Finally, and most importantly, the authors would like to thank all the patients who made this study possible.
REFERENCES
