Abstract
Background Education delivery in higher education institutions was severely affected by the COVID-19 pandemic, with emergency remote teaching developed and adapted promptly for the circumstances. This rapid review investigated the effectiveness of alternative education delivery strategies during the pandemic for medical, dental, nursing and pharmacy students, to help plan and adapt further education provision.
Methods We included 23 primary studies in undergraduate education, all published in 2020-2021, no relevant UK-based or postgraduate studies were found. Included studies comprised 10 single cohort descriptive; 11 comparative descriptive; and two RCTs. There was considerable variability in terms of students, type of distance learning, platforms used and outcome measures.
Results In medicine (n=14), self-reported competency and confidence, and demonstrable suturing skills were achieved through participating in remote learning. However, lower levels of knowledge were obtained by students who received virtual or blended learning compared to in-person teaching (low-very low confidence). Using bespoke interactive platforms in undergraduate medical training was superior to standard video (low confidence) or ‘textbook’ presentations (very low confidence).
In dentistry (n=2), remote learning led to knowledge gained (low confidence), but self-reported practical and interpersonal skills were lower with remote rather than in-person learning (very low confidence).
In nursing (n=3), remote learning, when compared to in-person, resulted in similar knowledge and self-reported competency levels (very low confidence) pre-COVID, but confidence was higher when learning or assessment was conducted virtually (low confidence).
In pharmacy (n=4), virtual learning was associated with higher skills, but lower knowledge compared to in-person, pre-COVID; self-reported competency and confidence scores were similar between the two groups (very low confidence).
Conclusions Remote teaching was valued, and learning was achieved, but the comparative effectiveness of virtual versus in-person teaching is less clear. Supplementary alternative or in-person practical sessions may be required post-emergency to address learning needs for some disadvantaged student groups.
Introduction
Education delivery in higher education institutions was severely affected by the COVID-19 pandemic, especially for healthcare students whose continuing education is imperative to maintain a well-educated healthcare workforce. Many courses transitioned to a period of remote emergency teaching,1–3 developed and adapted promptly for the circumstances, largely without prior contingency planning. For example the American Society of Plastic Surgeons announced free access to its online Education Network for all medical students with an interest in plastic surgery,4 whilst Ahmed at al5 suggested a range of online tools and resources that could be employed for online rheumatology education. In Jordan, distance e-learning was promptly engaged to maintain the continuity of medical education,6 and in Pakistan dental educators came up with innovative solutions to resume dental education remotely.7 Bakshi et al8 argued that whilst the COVID-19 outbreak disrupted the educational experiences of medical students worldwide, this was particularly significant in areas such as ophthalmology where structured education and clinical exposure had already declined. A shift to virtual education for nursing students in Iran highlighted some of the challenges faced by educators and students, such as lack of infrastructure, reduced readiness of educators and students for e-learning, and the time to prepare educational content,9 whilst educators in Canada10 emphasised the importance of continuing to engage nursing students online. Reviews have also highlighted the challenges in migrating to remote education11,12 which include poor knowledge of staff on how to deal with technology, poor internet connections and difficulty in transitioning content for online learning.11,12 By contrast, some students and staff report satisfaction with remote learning,2,13 especially when collaboration and engagement with peers is facilitated.2
A preliminary search of repositories specific to COVID-19 literature identified several existing reviews of alternative education delivery strategies for medical and healthcare students during the COVID-19 pandemic. The systematic review by Wilcha et al.3 looked at the effectiveness of virtual teaching for medical education and suggested that it was effective. However, searching was limited to two databases, including Google Scholar, and the review appears to have been conducted by one author with no critical appraisal conducted.3 Another systematic review by He et al.13 explored the use of synchronous distance education (videoconference or web conference, online classroom or virtual classroom) compared with traditional education for medical, dental, nurse, pharmacy students and other health science–related students.13 It found that there were no significant differences in terms of knowledge or skills but that satisfaction was rated higher for distance education.13 For nursing students, a scoping review by Jowsey et al.14 suggested that when delivered purposefully, blended learning (a mix of face-to-face and online study) can positively influence and impact on the achievements of students, especially when used to support distance education.14 However, none of the existing reviews specifically explored effectiveness of alternative education delivery strategies for medical, nursing, dental and pharmacy students, or allied health professionals during the COVID-19 pandemic, or provided a separate summary of the evidence for these disciplines. An initial scope of the evidence base for these healthcare disciplines identified a significant volume of primary research in the area for medical, nursing, dental and pharmacy students but very little for other healthcare disciplines including allied health professionals. We therefore conducted a rapid review of the effectiveness of alternative education delivery strategies that have been put into place for undergraduate and postgraduate medical, nursing, dental, and pharmacy students.
Methods
This rapid review was registered with the International prospective register of systematic reviews (PROSPERO) following the completion of the database searches, and study selection (Registration number: CRD42022304295).
Eligibility criteria
The inclusion criteria were informed by the PICO (Participants, Intervention, Comparison, Outcomes) framework (see Table 1).
Search strategy
Searches were conducted across four databases: On the OVID platform: MEDLINE and Embase, on the EBSCO platform: CINAHL and ERIC, from December 2019 to 8th June 2021 for English language citations. An initial search of MEDLINE was undertaken (medicine or medical or nurs* or dental or dentistry or pharmacy or pharmacist) AND education* or train* or teach* or student* or undergraduate* or postgraduate* AND COVID* or coronavirus) followed by analysis of the text words contained in the title and abstract, and of the index terms used to describe the article. This informed the development of search strategies tailored for each information source (additional material one). The reference list of all included studies was screened for additional studies.
Study selection process
All citations retrieved from the database searches were imported into EndNote™ and duplicates and irrelevant citations removed and then imported to Covidence™ for study selection. Two reviewers dual screened at least 20% of citations using the information provided in the title and abstract using the software package Covidence™, resolving all conflicts. The remaining citations were then screened by a single reviewer, screening with categories of ‘include’ and ‘exclude’. To streamline the review process, the project team decided against a third category of ‘unsure’ and instead, where there was uncertainty about a citation, it was categorised as ‘include’ and the decision was made based on the full text. The full texts were screened for inclusion by one reviewer using a purposefully designed form which was piloted using approximately 10 manuscripts. One reviewer then screened full text manuscripts, and another reviewer checked all excluded manuscripts.
Data extraction
All demographic data were extracted directly into tables by one reviewer and checked by another. The data included specific details about the interventions, populations, study methods and outcomes of significance to the review question and specific objectives. A template for the data extraction process was piloted on manuscripts for each of the included study designs before use. All outcome data were extracted directly into tables by one reviewer and checked by another.
Quality appraisal
The methodological quality of all the research studies was assessed by one reviewer, and judgements verified by a second reviewer, using JBI design-specific critical appraisal tools (https://jbi.global/critical-appraisal-tools). When a study met a criterion for inclusion a score of one was given. Where a particular point for inclusion was regarded as “unclear” it was given a score of zero. Where a particular point for inclusion was regarded as “not applicable” this point was deducted from the total score. All included studies were assessed using this method and their overall critical appraisal scores were calculated and are displayed for each study in Tables 2 and 3. For the full details of the critical appraisal scores see additional material two.
Synthesis
The data were reported narratively as a series of thematic summaries15 and presented separately for each health care discipline. Two RCTs were included in the review but there was insufficient homogeneity across the studies and therefore we were unable to perform a meta-analysis.
Assessment of body of evidence
The confidence in the synthesised findings was assessed by one reviewer and judgements verified by a second reviewer. The RCTs were assessed using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach.16 Due to heterogeneity of the different interventions within similar settings, outcome data were only available for results from single studies and guidance was followed on undertaking GRADE for data of this type.17 Quantitative descriptive studies were assessed by applying the principles of GRADE.18 For further details of this processes see additional material three and four. Most findings in this rapid review were of low or very low quality and ratings are displayed for each study in Tables 2 and 3. This was mainly due to imprecision because of small sample sizes, and/or confidence intervals not being reported, and/or limitations because baseline levels of the outcome of interest not being controlled for, and/or lack of clarity of confounding factors.
Results
Of the 10,978 citations retrieved from our searches, 21 descriptive studies and two RCTs met our eligibility criteria. For details of the excluded studies see additional material five. The included studies focused on undergraduate medical students (n=14), undergraduate dental students (n=2), undergraduate nursing students (n=3) and undergraduate pharmacy students (n=4). We did not find any studies that focused on postgraduate students, and research, that focused on clinically based postgraduate training, such as internships, were excluded. The flow of citations through each stage of the review process is displayed in a PRISMA flowchart, 19 see Figure 1.
Overview of evidence base for medical students
Six pre-test / post-test designs20–25 and six post-test only descriptive studies.26–30 and two RCTs,31,32 provided evidence of the effectiveness of alternative education delivery strategies for undergraduate medical students during the COVID-19 pandemic (see Table 4). Most studies (n=7) were conducted in the USA.21–25,29,33 The remaining studies were conducted in Germany,26,27,31 Japan,20 South Korea,28 Switzerland32 and Greece.30
These covered a wide range of both university and clinical based modules/ courses and included neurosurgery,21 surgical instruments, knot tying and suturing,24 digital histology,26 a residency preparation course,22 simulated patient consultations, documentation, and case presentation,27 simulated clinical experience in respiratory unit and general medicine,20 generic medical education,28 neuroanatomy,29 emergency medicine25,33 musculoskeletal system anatomy and neuroanatomy,30 the National Institutes of Health Stroke Scale,32 operative techniques and skills,31 and informed consent for surgical procedures.23
A variety of different online platforms was used to deliver synchronous learning; five used the Zoom video conferencing platform20,21,24,26,27 three used the University Supported Management Systems: CANVAS22,29 or Meducator,30 one used Microsoft teams,33 another Skype for business,30 and three did not specify the type of video communication software used.23,25,28 Other methods included neuroanatomical interactive virtual activities using “Digital Neuroanatomy” software,29 simulated patient encounters employing online MedEd Case X videos,33 and structural specimens replaced by photographs.30 Five studies also incorporated asynchronous elements using pre-recorded lectures23,28,30 or readily available podcasts.25,33 For one further study the course content (8 topics) was organised by 12 rising1 fourth-year medical students under supervision.25 The two RCTs used bespoke interactive online platforms31,32 and compared the outcomes to those students learning the same topic via a standard video format31 or textbook based preparation.32
Studies were conducted with students in their final year (Clerkship / Interns) (n=7),20,22,24,25,27,32,33 first year (n=2),29,30 second and third years (n=1),26 third year only (n=1),23 across all years (n=1),21 and a further two did not specify the year of study.28,31 Outcomes explored were confidence (n= 5),21,22,24,25,27 competency (n=2)20,23 and knowledge (n=6).26,28–30,32,33
Levels of competency, confidence and knot tying and suturing skills were found to have improved across the course of learning and a further study suggested that levels of competency were the same when learning was conducted virtually (2020) compared to in-person pre-COVID (2019). Evidence from RCTs showed that knowledge was greater when learning was conducted using bespoke interactive platforms with a standard video format reported during the COVID pandemic. Evidence from descriptive studies showed mixed results for knowledge, assessed and compared between cohorts at the end of virtual learning (2020) and in-person learning (2019). Four studies reported lower levels of knowledge for students in the virtual cohort and one further study found no difference.
Overview of the evidence base for dental students
Two post-test descriptive studies (see Table 5) conducted in Poland34 and Germany35 provided evidence of the effectiveness of alternative education delivery strategies for undergraduate dental students studying specific modules or courses in conservative dentistry with endodontics34 or operative dentistry35 during the COVID-19 pandemic. In one study, the teaching consisted of asynchronous online screencasts (screen-captured PowerPoint presentations with narrated audio) using Stud-IP (a source learning management system) and discussed via synchronous video meetings using the Zoom video videoconferencing platform.35 The other study used a blended learning approach using the Blackboard Collaborate platform.34 The outcome of interest across both studies was knowledge acquisition. Findings from both studies suggest that these alternative educational methods contributed towards knowledge and skill acquisition assessed through a self-assessment survey completed by fourth year students34 and through a final summative examination.35 However, the evidence suggests lower levels of knowledge for the subtopic of periodontology and lower levels of practical skills for 3rd year dental students when learning was conducted virtually compared to in-person.
Overview of evidence base for nursing students
Three descriptive studies (see Table 6) conducted in Spain,36 Japan37 and USA38 provided evidence for the effectiveness of alternative educational delivery strategies for nursing students studying a specific module in human genomics,37 simulation in paediatric clinical practice38 and for the delivery of remote OSCEs with COPD patients36 during the COVID-19 pandemic. All three studies compared a group of students receiving a remotely delivered educational package with a group receiving standard, in-person education. In two studies the comparison groups were students from the previous, pre-COVID academic year, however, Weston and Zauche38 studied a cohort of students from the same academic year, 2019-2020, where half had received the standard educational package before the alternative version was introduced. Only one study used a pre-test / post-test design and thus compared results within as well as between groups.37 In this study, the conventional course was transferred to remote synchronous learning (narrative over PowerPoint), also uploading handouts and worksheets with no changes to content.37 Arrogante et al.36 used the virtual classroom platform Blackboard Collaborate to conduct OSCEs comprising eight simulated clinical scenarios with standardised patients. Weston and Zauche38 substituted virtual simulation using the i-Human platform to replace in-person clinical practice and simulation laboratory learning. Outcomes explored were competency (n=2)36,37 confidence (n=1),37 and knowledge (n=2).37,38
The evidence suggests that levels of competency were the same and levels of confidence were higher when learning or assessment was conducted virtually (2020) compared to in-person pre-COVID (2019). Knowledge improves regardless of whether the learning has been conducted virtually (2020) or in-person pre-COVID (2019).
Overview of the evidence base for pharmacy students
Four descriptive studies (see Table 7), all conducted in the USA, provided evidence for the effectiveness of alternative education delivery strategies for undergraduate pharmacy students studying specific modules or courses in integrated patient care,39 hypertension/drug information,40 advanced pharmacy experience,41 delivery of remote Objective Structured Clinical Examinations (OSCEs) for patient counselling, and taking a medical history42 during the COVID-19 pandemic. Two studies used a pre-test/post-test design,40,41 the remaining two reported a post-test only study design, with a comparison between the study population and an earlier (pre-COVID) cohort of students.39,42
In one study the teaching included remote synchronous learning,41 three studies used the Zoom videoconferencing platform,39,41,42 two studies used the University platform Blackboard Collaborate40 and one study also used the University Supported Management System: CANVAS.41 The outcomes of interest that were explored were competency (n=2),39,40 confidence (n=2),40,41 knowledge (n=2),39,41 skills (n=2)41,42
Evidence suggests competency outcomes improved across the course of learning and were similar when learning was conducted virtually (2020) compared to in-person pre-COVID (2019). Confidence was found to either improve across the course of learning or be the same for virtual (2020) compared to in-person pre-COVID (2019) learning. However, lower levels of knowledge were reported when learning was conducted virtually compared to in-person pre-COVID. The evidence suggests that, overall, students performed similarly between in-person (2019) and online (2020) OSCEs, although for some, skills performance was higher when students undertook these virtually.
Discussion
The findings of this rapid review are based on very limited evidence for dental (2 descriptive studies), pharmacy (4 descriptive studies) and nursing (3 descriptive studies) education. Only one finding from across all twelve of the descriptive studies that focused on medical education was rated as being of moderate quality. As expected, levels of knowledge, competency and confidence improved over the course of virtual learning. However, when results were compared to students who had completed in-person learning in the years before the COVID-19 pandemic, results were mixed. Most studies across the disciplines reported similar findings across all outcome variables suggesting that virtual learning produced similar results to in-person learning. To our knowledge this is the first rapid review of the effectiveness of alternative education delivery strategies for undergraduate and postgraduate medical, dental, nursing and pharmacy education during the COVID-19 pandemic.
Previous systematic reviews showed online learning outcomes to be comparable to in-person learning. At the time of conducting this rapid review we were unable to locate any reviews that took an interdisciplinary approach. Given the potential overlap and value in sharing practices across the various healthcare educational contexts, we aimed to address this gap.
Evidence from two RCTs showed that knowledge was greater when learning was conducted using bespoke interactive platforms compared with non-interactive formats, reported during the COVID pandemic.31,32 These findings concur with research conducted in the field prior to COVID-19, with three systematic reviews suggesting that pre-planned online eLearning for undergraduates in health professions is equivalent, possibly superior to traditional learning.43–45
Data from this rapid review indicated that the transition from traditional teaching into remote methods seemed to affect students’ performance at exams, particularly so for the practical based subjects in dentistry and medicine. It is recognised that emergency remote teaching and learning differs from planned online learning.46,47 Most remote teaching and learning that initially took place during the COVID-19 pandemic was not planned and was adapted promptly due to the emergency circumstances that presented. In addition, this new learning did not take into account the additional stress that e-learning can cause48 or incorporate strategies to increase social presence which Natajaran and Joseph49 argue is essential to improve student nurses’ satisfaction with online teaching.
Implications for policy and practice
Healthcare educators need to revisit the research base surrounding remote learning and consider this evidence when planning future online education. Whilst lessons learnt were quickly put into place, the COVID-19 pandemic brought issues to the fore that have long been debated in healthcare education: reduced clinical exposure, a move away from mass didactic education, and the need to ensure all healthcare students are provided with the skills and knowledge required to transition to competent caring health professionals with the ability to think critically and source and apply evidence to practice. With the increasing need for skilled healthcare professionals, policy makers need to consider how educational institutions can be provided with the resources required and how existing educators can be upskilled and supported to develop technology-enhanced learning experiences. Students from school entry age onwards need to be prepared for more online and blended learning experience which should include providing them with strategies they can use to support their emotional and psychological well-being, whilst accessing remote learning. Future research should investigate the effectiveness of blended learning approaches compared to more traditional education, in addition to investigating the views and perceptions of both students and educators and the barriers and facilitators to engaging effectively in blended learning.
Limitations
To complete the review rapidly a limited number of databases were searched, and further studies may have been identified if additional bibliographic databases had been used. Out of the 23 included studies none was conducted within the UK and the majority (n=21) were descriptive studies. All included studies focused on undergraduate not postgraduate education. Of these, 11 studies employed a pre-test/post-test design, and the remainder were post-test only evaluations. The two RCTs both used a test or examination to assess knowledge, but these evaluated two different interventions and therefore statistical pooling of data using meta-analysis was not appropriate. Furthermore, both studies had small sample sizes and poor response rates (75/158 and 44/58).
Regarding the limitations of this review’s methods, the tool used for evaluating the confidence of the quantitative descriptive studies is an adaptation of GRADE and has not been approved by the tool’s originators. Finding well conducted comparative research proved challenging as not all educational researchers sign up to this experimental ideology when it comes to investigating teaching. Indeed, most published educational studies are small scale and qualitative in nature. There is, however, an agreement that there is a lack of high-quality studies to serve as models for future development in remote learning and teaching.50,51 We therefore suggest that studies that do apply the experimental approach should aim to enhance their research rigour in order for them to provide findings that can be synthesised more meaningfully. We also recognise the potential impact of the pandemic on resources and time, all of which would have likely impacted the quality of research. For this reason, we suggest that our rapid review provides a platform for further research that will consider the large body of literature that has emerged from the various fields of healthcare education since we conducted our review.
Conclusions
Remote teaching was valued, and learning was achieved, but the comparative effectiveness of virtual versus in-person teaching delivered in a pandemic is less clear. In addition, the available evidence is insufficient to demonstrate equivalence for student speciality groups and it is unclear whether planned remote teaching, rather than relying on emergency adaptation, would be more effective. For some healthcare students, academic achievement appears to decline when practical learning is insufficient, and this is something that must be addressed. However, this could be attributed to the sudden transition to online learning mid semester in which students did not have a chance to prepare or plan how they may need to adjust their own learning strategies. Moreover, teaching online requires a new skill set and educators may have had very little chance to upskill. It is therefore difficult to use the findings to inform future educational planning. Identifying which aspects of health education delivery are best delivered via a particular format or platform will be key to improving the efficiency of learning for organisations and accessibility of material for students. Time will tell as to the career progress of the students whose studies have been affected by COVID-19 with educators and regulators ensuring that health care professionals are supported in their learning and standards are maintained. Further research with robust methods to evaluate alternative education delivery strategies is needed to inform policy decision-making in this area.
Data Availability
All data produced in the present work are contained in the manuscript
Data availability statement
No data are associated with this article.
Competing interests
The authors declare they have no conflicts of interest to report.
Grant information
The Wales Centre for Evidence Based Care was funded for this work by the Wales COVID-19 Evidence Centre, itself funded by Health & Care Research Wales on behalf of Welsh Government.
Extended data
Additional material one: Full search strategies http://www.primecentre.wales/resources/RR/Clean/RR00004_Supplementary_information_Healthcare_education.pdf
Additional material two Critical appraisal scores http://www.primecentre.wales/resources/RR/Clean/RR00004_Supplementary_information_Healthcare_education.pdf
Additional material three: Tool for assessing the confidence of synthesised findings from quantitative descriptive studies http://www.primecentre.wales/resources/RR/Clean/RR00004_Supplementary_information_Healthcare_education.pdf
Additional material four: Evaluation of confidence using GRADE http://www.primecentre.wales/resources/RR/Clean/RR00004_Supplementary_information_Healthcare_education.pdf
Additional material; five: excluded studies http://www.primecentre.wales/resources/RR/Clean/RR00004_Supplementary_information_Healthcare_education.pdf
Acknowledgements
The authors would like to thank Assim Javaid for his contribution during stakeholder meetings to guide the focus of the review and interpret findings. In addition, thanks to Professor Jane Noyes for information regarding the adaption of the GRADE approach for quantitative descriptive studies. The authors would also like to express their gratitude to Maggie Hendry and acknowledge her contribution to the Wales COVID-19 Evidence Centre report. Maggie Hendry participated in study selection, data extraction, and quality assessment.
Footnotes
↵1 In the summer of an academic year, there are two “senior” classes (these are fourth year college students in America). The class that just graduated, known as graduating seniors, and the one that will be seniors, when fall comes around known as “oncoming senior” or “rising seniors.”