A Light-weight Text Summarizer for Fast Access to Medical Evidence

Introduction

Current clinical practice guidelines urge practitioners to follow the principles of evidence-based medicine, which requires them to integrate the best external scientific evidence with clinical expertise.^1,2 Early and recent studies show that one of the biggest obstacles to evidence-based medicine practice is information overload caused by the large volume of medical literature available.³ Searching through medical evidence is too time consuming and practitioners often consider the task to be unproductive.⁴ A standard clinical query on PubMed,¹ which indexes over 30 million articles, typically returns multiple pages of research publications. Hersh et al.⁵ discussed that it takes more than 30 minutes for a practitioner to search for an evidence-based answer, and, particularly at point-of-care, practitioners cannot afford to spend that much time. Literature searching, and fast access to relevant information, is particularly beneficial for medical students and young practitioners because of their lack of clinical experience.

Motivated by the importance of incorporating scientific evidence in everyday medicine practice, automated text summarization research has attempted to address the problem. In particular, query-focused text summarization approaches have been explored to aid evidence-based medicine.^6–8 These systems take queries (in natural language or key-terms) as input and output query-relevant summaries. In terms of automatic summary qualities, the performances of successful approaches designed for the medical domain have relied heavily on domain-specific knowledge sources.⁹ For example, one study Demner-Fushman and Lin¹⁰ incorporated sentence-level knowledge in a supervised classification system trained to detect outcome sentences. Sarker et al.⁸ and ShafieiBavani⁷ utilized manually annotated summarization datasets specific to the domain to generate extractive and abstractive summaries—both systems relying heavily on the identification of domain-specific generalizations, concepts, and associations. Hristovski et al.¹¹ proposed the use of domain-specific semantic relations for performing question answering for biomedical literature. Further discussion of such systems is outside the scope of this short paper, and detailed descriptions of medical text summarization systems over the years are available through survey papers.^12,13

Adaptation of summarization systems to a particular domain can be computationally expensive, and require large numbers of external tools.¹⁴ Within the medical domain, systems typically use the Unified Medical Language System,² and tools such as MetaMap¹⁵ are employed to derive domain-specific knowledge from lexical representations. This is in turn used in downstream tasks, or as features in learning systems. Heavy dependence on these domain-specific resources introduces disadvantages: (i) the systems are not very portable, (ii) they are difficult to re-implement and/or deploy without the knowledge sources, and (iii) the systems are slow and resource-heavy.

The goal of our work is to design a light-weight and fast medical text summarization system that is decoupled from knowledge bases. In our view, such a system may be operationalized as a web-based service, enabling practitioners to review medical evidence at point-of-care. The proposed system relies on publicly available labeled and unlabeled data, dense word vectors learned from the unlabeled data, and a set of simple features that require little computational resources and time. Automatic evaluation of our system with a state-of-the-art system on a standard dataset shows that our approach produces comparable summary quality.

Methods

We use a public dataset—the corpus provided by Molla-Aliod et al.,¹⁶ which contains a total of 456 questions along with expert-authored single- and multi-document evidence-based summarized responses to them. Each question is generally associated with multiple single-document summaries, which present evidence from distinct studies. The abstracts of the studies from which the answers were derived are made available from PubMed. In total, the corpus contains 2,707 single-document summaries. To ensure fair comparison, we obtained the exact train-test split from the authors of the QSpec system⁸ (1,388 for training, 1,319 for evaluation). The training set is used to devise feature scoring techniques and learn weights for all the feature scores. The summary sentences are scored as the sum of the weighted feature scores, and chosen sequentially (from first to last), taking into consideration the target sentence position and the contents of the chosen sentences. The scoring process can be summarized as: where is the score for sentence number m of a text, given the summary target sentence number t_n, and w_i and f_i are the weight and score for feature i, respectively. For each t_n, the top-scoring sentence is chosen. Following experimentation with multiple features, we selected five that could be computed fast, and proved to be useful when used in combination. We describe these in the following paragraphs.

Sentence Length Score

Sentence length is a metric that may filter out uninformative, short sentences by assigning them a lower score, while rewarding sentences that are relatively longer in a document. In past research in this domain, it has been mentioned to be particularly effective for tie-breaking. In our work, we attempted to assign penalties to very short sentences (e.g., 1—3 word sentences), which often represent section headers. At the same time, our goal was to assign higher scores to longer sentences---with decreasing gradients for very long sentences, such that this score does not play a significant role in choosing between those informative sentences.

Our experiments on the training set suggested that a sin function conveniently served this purpose. The average sentence length in the training data is approximately 150 characters, so we considered 0 and 300 characters to be the lower and upper length limits, respectively, and mapped the lengths to the range [, ]. Following that, we applied a sin function to the mapped value to generate a length score between [-1,1] (Figure 1).

• Download figure
• Open in new tab

Figure 1

Clockwise from top-left: sine function for sentence length score, maxed at 300 characters; first, middle and last sentence relative position distributions from the best-scoring extractive training set summaries.

Evaluation, Results and Discussion

As mentioned, we used the ROUGE summary evaluation tool to compare the performance of our system with other systems. Since the summaries are not length-restricted, we used the ROUGE-L F₁-score as our evaluation metric. ROUGE scores have been shown to be highly correlated with human evaluators.²¹ Table 1 presents the performance of our system along with several other systems, including the state-of-the-art for this task (QSpec). Identical training-test splits were used for evaluation. The table shows that despite the simplicity of our approach, its performance is comparable to the state-of-the-art, and significantly better than other baselines.

Due to the simplicity of our approach, it can be easily re-implemented, customized or extended for real-life settings, and the results can be reproduced without requiring the use of third-party tools. Compared to the resource-heavy QSpec system, which requires query and sentence classification, and the generation of UMLS semantic types and associations, our approach requires minimal preprocessing. Only a set of pre-trained word embeddings are required. The light-weight nature of the summarizer also means that it runs faster than QSpec. On a standard computer (Intel® i5 2.0 GHz processor), it takes our summarizer a few minutes to summarize all the documents in the test set.

We obtained the word embeddings from past research and used them without modification. There is a possibility that the learning of these embeddings can be customized to the summarization task for improving performance. Furthermore, since the embeddings can be learned from publicly available data, and no other resources are required by the summarizer, the system can be implemented anywhere with minimal effort.

Conclusions and Future Work

In this paper we presented a simple, query-focused, extractive summarization system suited for application in the domain of evidence-based medicine. Unlike past systems in this domain that rely heavily on domain-specific resources such as knowledge bases, our system is light-weight in nature and relies only on distributed representations of words learned from unlabeled text. Using a set of similarity-based and structural features, our system performs comparably to the state-of-the-art system, with a ROUGE-L F₁-score of 0.166. Because of the simplicity of our system, it can easily be ported and implemented in different settings.