Effectiveness of the EPA-based ‘Toolbox Family Medicine’ on students’ learning satisfaction: study protocol for a controlled trial

Background and rationale {6a}

Family practices constitute an important learning environment for medical students (Pershing et al., 2013). However, practice rotations are heterogeneous teaching situations that students often find underwhelming (Herwig et al., 2017). The extent of students’ participation varies considerably, although it significantly affects students’ interest in primary care careers (Herwig et al., 2017). There are several reasons for unsatisfactory practice rotations: Supervising physicians balance competing tasks as they provide teaching integrated into their daily patient services. In addition, time pressure and lack of didactic training impair teaching quality.

Aiming at a better and more standardized learning environment in family practices, we developed the so-called ‘Toolbox Family Medicine’ (TFM). It provides standardized learning content suitable for students’ levels and allows teaching adaptable to actual practice conditions. Learning content is structured following the concept of entrustable professional activities (EPA). All EPAs describe small tasks typical for routine primary care that students have to perform guided by the respective teaching physician (Ten Cate et al., 2015). Completing an EPA, students receive feedback from the supervisor on their performance, which provides the basis for future entrustment decisions. This linkage of activity and supervision level creates a performance-based training, so-called competency-based medical education. The overall aim is to gradually reduce supervision and increase students’ responsibilities based on individual progress (Peters et al., 2017). The five grades described by Chen (2015) are “allowed to observe” (grade 1), “direct supervision” (grade 2), “indirect supervision” (grade 3), “acting unsupervised” (grade 4), “guiding others” (Grade 5) (see Table 1).

Starting in Canada, this concept of competency-based medical education has already been implemented in several countries as a didactic approach for the graduate level. In Germany, EPAs have just started to become part of medical undergraduate and graduate education. Academic teachers of the Charité Berlin University defined twelve core EPAs, grouped into 5 EPA domains, to measure preparedness for entry into residency: 1. along the clinical encounter, 2. general medical procedures, 3. communicating with patients, 4. communicating and collaborating with colleagues, 5. patient care in special situations (Holzhausen et al., 2019). A pilot study described the implementation of EPAs with 62 final year students in four German universities concluding that more work is needed to integrate EPAs in medical curricula (Schick et al., 2019).

Our TFM is an approach to implement EPAs in general practice rotations of undergraduate students. Students and supervisors will be surveyed directly after the rotation. Students’ overall satisfaction with their learning progress is the primary outcome of this controlled trial.

Objectives {7}

This study evaluates the effectiveness of the teaching concept TFM on students’ overall satisfaction with their learning progress after the rotation. The evaluation will compare students from practices with TFM use (intervention) and without (control). We hypothesize that the TFM will improve learning satisfaction compared to traditional training.

Trial design {8}

This teaching quality improvement project will be performed as an intervention trial. Teaching practices and students will be allocated in a groupwise manner. The first 47 students will serve as the control group. The next 47 students will serve as the intervention group using the toolbox.

This approach is needed to avoid contamination on the student or practice level, i.e. a practice will not educate control group students after participating in the toolbox teaching. All control practices will receive the toolbox education and materials after the study’s completion (waiting list control approach).

Study setting {9}

The study will be conducted in family practices accredited by the Institute of General Practice and Family Medicine, University of Bonn. They are located in the greater Bonn area. As part of the undergraduate curriculum, medical students have a two-week practice rotation in family medicine, which takes place in their last clinical semester prior to the final year. All practices and students of the next two terms will be included in the quality improvement project.

Eligibility criteria {10}

Inclusion and exclusion criteria for practices: Family practices are eligible if they are accredited teaching practices of the Institute of General Practice and Family Medicine, University of Bonn.

Inclusion and exclusion criteria for students: Participating students must be enrolled at the University of Bonn and scheduled for the family practice rotation. Students will be excluded when changing the practice during their rotation.

Who will take informed consent? {26a}

The Ethics Committee of the Medical Faculty of the University of Bonn has agreed to the teaching quality improvement project and deemed it exempt from review on August 26, 2021. Teaching practices and students participate on a voluntary basis.

Additional consent provisions for collection and use of participant data and biological specimens {26b}

Not applicable, because no biological specimen will be obtained.

Explanation for the choice of comparators {6b}

Students of the control group will complete the usual general practice rotation in practices without TFM intervention.

Intervention description {11a}

The intervention ‘Toolbox Family Medicine’ comprises (1) a TFM training for teaching physicians focused on didactics, (2) the Toolbox Family Medicine with the EPAs and (3) an EPA-compendium, recording the performed EPAs with the respective supervision level (see Table 1).

The TFM intervention will be applied in a two-week rotation in intervention practices.

Elements of the ‘Toolbox Family Medicine’ (TFM):

1. TFM training for teaching physicians:

   1. Invitation email with a brief explanation of the new concept.

   2. Practice visits to hand over and introduce the TFM card box.

   3. One-hour online workshops for supervising physicians: Groups of six to ten teaching physicians will be educated on the concept and use of the toolbox focusing on didactics (e.g. feedback and teachable moments).

2. Toolbox with EPAs (index card box):

   1. Index cards with information on EPAs, feedback rules, application of the toolbox.

   2. Index cards with short and more complex tasks supplemented by the recommended supervision levels, more in-depth information and questions.

Following the supervision levels for medical education (see Table 1) (Chen et al., 2015), we defined limits for the performance of the various medical activities for undergraduate medical education (see Table 2). During their two-week rotation, students may perform EPAs up to a level of 3a.

An expert panel of five experienced supervising physicians and three GPs in training defined a total of 72 EPAs in five fields: medical history (1), physical examination (2), skills (3), communication (4), chief complaints (5). For details see Table 2.

For the two-week, full-day rotation in family practices, 21 EPAs were selected as core EPAs obligatory for all students. The teaching physician may select additional EPAs. After performing an EPA, the teaching physician will provide feedback to the student on the entrusted level. Additional information and questions on the index cards provide an orientation for the teaching physician for more in-depth teaching related to the EPA. Students should reach an entrustment level of 2b or 3a on all core EPAs, as defined in Table 2. Hence, respective EPAs may be performed and supervised several times until the student reaches the required entrustment level (Peters et al., 2017).

Criteria for discontinuing or modifying allocated interventions {11b}

Teaching physicians who are invited to the intervention but unable to attend the scheduled TFM training are assigned to the control group. The statistical analyses will account for this.

Strategies to improve adherence to interventions {11c}

Practices and students are routinely provided with the contact information of the teaching secretary (phone number, email) to report any problem prior or during the rotation.

Relevant concomitant care permitted or prohibited during the trial {11d}

Not applicable as the study is a teaching intervention.

Provisions for post-trial care {30}

Not applicable as the study is a teaching intervention.

Outcomes {12}

Participant timeline {13}

The timeline for teaching physicians and students is outlined in Figure 1.

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 1:

Schedule of enrolment, interventions, and assessments.

Sample size {14}

The sample size was calculated based on the primary outcome (students’ overall learning satisfaction). The primary outcome will be measured according to German university grades ranging from 1 (best) to 5 (worst). The null hypothesis is that the median category of the primary outcome is equal between control and intervention groups. The Wilcoxon-Mann-Whitney-Test was used to determine the sample size based on the approach of Happ (2019). This method requires prior information about the control and intervention groups, which was retrieved from the study of Herwig et. al. (2017). For the control group, we assumed the absolute frequencies of the group “interest in primary care decreased” for the overall rating of the clerkship as prior information. The term “prior difference” is defined as difference between absolute frequencies of the primary outcome in the “increased” and “decreased” groups in overall rating of clerkship. We assumed half of the prior difference as a minimally relevant educational effect that we are interested in detecting. Based on this assumption, the prior sample of the intervention group was created by adding half of the prior difference to the prior sample of the control group. Furthermore, a significance threshold alpha of 5 %, power level of 80 %, and a student dropout of 5 % is applied to determine the sample size. As a result, this study aims for a controlled trial with at least 94 medical students (47 intervention, 47 control).

Recruitment {15}

Scheduling for rotations will take place as usual. Practice rotations are part of the compulsory medical school curriculum. Family practices are invited by email and by phone. A complete timeline is presented in Figure 1.

Sequence generation {16a}

Using the routine academic schedule for upcoming rotations in family practices, half of the students and respective practices will serve as the control group. The second half of the students and respective practices will serve as the intervention group. Teaching physicians of the intervention group will be trained successively in small groups before new students are assigned to their practices. Practices participate based on availability.

{16b}

The allocation sequence is based on the academic year schedule for practices and students. No additional allocation mechanisms are needed.

Implementation {16c}

Enrollment and assignment of students to practices will be performed by the teaching office as usual. After reaching the planned number of control practices, the subsequent practices will serve as intervention practices.

Who will be blinded {17a}

No blinding. Students and physicians of the intervention group are aware of the toolbox usage.

Procedure for unblinding if needed {17b}

Not applicable.

Plans for assessment and collection of outcomes {18a}

Finishing their two-week rotation students will be invited to complete a survey on eCampus, the teaching platform of the University of Bonn. Teaching physicians will be invited to evaluate each rotation using a professional survey platform following data protection laws.

Plans to promote participant retention and complete follow-up {18b}

Physicians and students will receive up to two reminders to promote follow-up.

Data management {19}

Plans for collection, laboratory evaluation and storage of biological specimens for genetic or molecular analysis in this trial/future use {33}

Not applicable. No samples collected.

Statistical methods for primary and secondary outcomes {20a}

This protocol contains the entire statistical plan. Data management will follow standardized procedures (SOPs, Standard Operating Procedures). Primary and secondary outcomes will be analyzed using descriptive statistics. Inferential statistics will be performed for the primary outcome and selected secondary outcomes to analyze for predictors of learning satisfaction.

Interim analyses {21b}

No interim analysis is planned.

Methods for additional analyses (e.g. subgroup analyses) {20b}

Subgroup analysis will be performed by practice clusters.

Methods in analysis to handle protocol non-adherence and any statistical methods to handle missing data {20c}

Data imputation with standard procedures will be performed if needed to handle missing data (Van Buuren S, 2018).

Plans to give access to the full protocol, participant level-data and statistical code {31c}

The datasets generated and analysed during the study will not publicly available due to confidentiality issues for participating students and teaching physicians but are available from the corresponding author on reasonable request.

Composition of the coordinating centre and trial steering committee {5d}

Not applicable, as this is a teaching quality improvement project.

Composition of the data monitoring committee, its role and reporting structure {21a}

Not applicable, as this is a teaching quality improvement project.

Adverse event reporting and harms {22}

Not applicable, as this is a teaching quality improvement project.

Frequency and plans for auditing trial conduct {23}

Not applicable, as this is a teaching quality improvement project.

Plans for communicating important protocol amendments to relevant parties (e.g. trial participants, ethical committees) {25}

In case of modifications of the protocol, the above-mentioned ethics committee will be informed.