ABSTRACT
Background The widespread adoption of electronic health records (EHRs) has resulted in the generation of large volumes of clinical notes. Learning algorithms and large language models (LLMs) train from these resources but are susceptible to noise—irrelevant or non-informative data from them. This sensitivity can lead to significant challenges, including performance degradation and the generation of inaccurate predictions or “hallucinations.” This study addresses a critical challenge in clinical informatics: efficiently filtering millions of documents for relevance before advanced language model processing, particularly in resource-constrained environments. We present a novel framework for determining document relevance in clinical settings, utilizing a chronic obstructive pulmonary disease (COPD) dataset.
Methods We developed a novel framework using weak supervision and domain-expert heuristics to generate “silver standard” labels for training data and expert annotated labels (gold stand),creating two datasets to optimize the model during the development phase and subsequent testing phase. Various text representation techniques, including Bag-of-Words, TF-IDF, lightweight document embeddings, compression-based features, and UMLS concept extraction, were evaluated. These representations were used to train Random Forest, XGBoost, and K-Nearest Neighbors classifiers. Models were optimized on a small expert-annotated dataset and evaluated on a held-out test set.
Results The combination of lightweight document embedding with a Random Forest classifier demonstrated the best performance, achieving a precision of 0.75, recall of 0.89, and F1-score of 0.81 (95% CI: 0.76-0.87) for identifying relevant COPD documents. This significantly outperformed baseline heuristics (precision: 0.70, recall: 0.38, F1-score: 0.50, 95% CI: 0.43-0.56) and other tested methods.
Conclusion Our study presents a novel framework for identifying COPD-relevant clinical documents using lightweight embedding and machine learning. This approach effectively filters pertinent documents, enhancing information retrieval precision. The framework’s scalability and minimal annotation needs make it promising for diverse healthcare applications, potentially optimizing clinical outcomes through efficient document selection for data-driven decision support systems.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study was funded by NIH
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study was approved by the Vanderbilt University Medical Center (VUMC) Institutional Review Board.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors
