A cross sectional survey of Australian and New Zealand specialist trainees’ research experiences and outputs.

Objective : To explore medical trainees ’ experiences and views concerning college-mandated research projects. Setting : Online survey (Apr-Dec 2021) of current and recent past trainees of Australian and New Zealand colleges recruited through 11 principal colleges and snowballing. Participants: Current trainee or completed training in the past 5 years. Main outcome measures: We asked participants: whether they were required to conduct research as part of their college training, how they conducted their research, and their research activity after training. Respondents were invited to submit project reports for reporting and methodological quality evaluation. Data were analysed descriptively. Results: Of the 372 respondents, 313 (86%) were required to complete one or more projects. Of the 177 who had completed their project (representing 267 projects), 76 provided information on 92 studies, with 34 reports submitted for evaluation. Most respondents developed their own research questions, study design and protocol, and conducted research in their own time, with 56% (38/68) stating they had the skills to complete their project. Most project teams consisted of their own medical specialty followed by statisticians, but seldom others.


Contributions, guarantor information and acknowledgements.
All authors agree with the viewpoints expressed in this manuscript.PS DH CB and CN conceptualised the study.PS, CW, RB, AB, CB, CN, PG, IS, ABB, MM, GK, HJ, EV, KF, DP, SM, DH as well as David Ellwood, David Pearson, Rhea Liang, Gordon Wright all contributed to the development of the methods and funding acquisition.PS developed the formal analysis plan (with guidance from AB), developed the data extraction tools, data curation processes, validation processes, conducted the data wrangling and formal analysis, developed the analytic and visualisation code, and oversaw project administration.She is the study guarantor.All listed authors contributed to dissemination of the survey, which was managed by Iris Gerke and overseen by PS.All authors contributed to the data extraction of uploaded files.We would like to acknowledge Tammy Hoffman and Mina Bakhit for their advice on using reporting guidelines, and Joanne Hilder for conducting the content analysis.
PS developed the original draft of this manuscript, DH CB and AB provided initial critical reviews, and all authors reviewed and approved the final version of this manuscript.
The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Summary box
Majority of medical specialty trainees are required to conduct a research project to develop their research skills.
We found the learning experiences are inconsistent, and the quality of research produced even more so.
A new approach is required that is tailored to the research skills required by most practicing clinicians, namely being expert in applying research to practice and in participating in collaborative research.Those wishing to become leaders in research should be supported to do so via a specialised well-supported pathway.

Background
Medical specialty training colleges often require doctors to conduct research to earn their professional qualifications.Such practices are widespread, including in the UK, Europe, Northern America, Africa, Asia (1)(2)(3)(4)(5)(6) and Australia.(7,8) In Australia, we found that college curricula for research focus on individuals conducting their own projects and generating publications, rather than research skills development and expert supervision.(8) This current approach may encourage rushed, poor quality, small-scale projects, and trainees may fail to learn how high quality research can contribute to patient care.(9)A review of ten surgical programs in the UK highlighted similar concerns and the authors questioned the quality of the research outputs and trainee experiences.(10) While this long-established approach to developing research skills has been questioned, (11)(12)(13) little investigation has been done to characterise the corresponding trainee research experiences and outputs.To address this information gap, we surveyed medical specialty trainees in Australia and Aotearoa/New Zealand about research activities performed as part of college training requirements.Specifically, we aimed to understand how often trainees are required to conduct research projects, how they conducted such studies, and their general views on the value of these activities.We also assessed the quality of design and reporting of their submitted research reports.

Methods
Between April and December 2021, we conducted an anonymous cross-sectional survey of current and past medical specialty trainees.We use the CHERRIES reporting guidelines for e-surveys to report our findings.(14) Recruitment materials, full survey, and analytic code (including packages) are available on our OFS website(https://osf.io/346xe/).

Recruitment.
We worked with 11 medical specialty colleges to disseminate the survey via newsletters or direct email.Additional recruitment strategies included direct email with potential participants through known contacts, invitation slides at various conferences and forums, and social media posts.
. Potential participants were provided with a survey link and were encouraged to share the link with eligible colleagues.
The survey was distributed via an anonymous link, meaning we were unable to track which recruitment method resulted in survey participation; and were not able to provide response rates.
We therefore report the number that started the survey and those who contributed to each question.

Survey content.
The survey contained three sections: a main survey (investigator developed) and two optional sections using validated instruments.We used a secure survey platform (Qualtrics( 15)), with builtin survey logic so participants only saw questions relevant to them (Supplementary File 1).
The main survey section was developed by a core group of authors and was informed by the literature on known problem areas in research (16) and how best to support trainee research in the workplace.(17) It was tested for face and content validity with the wider authorship group which included experts in medical education, clinical research, research waste and Evidence-Based Practice (EBP), and representation across a wide range of specialties, and piloted with team members who included potentially eligible participants and trainee supervisors.
Participants were asked when and where they completed their most recent specialty training, their views on the importance of conducting research during specialty training, and how many projects they had completed (if any).We defined a project as any project type work that was mandated by the college as part of specialty training, including primary research, secondary research (e.g.: systematic reviews), audits and quality improvement projects.
For each project, we asked respondents: how they formulated their question; whether they performed a literature review or developed a protocol prior to commencing their project; the skills mix on the project team; access to relevant expertise and supervision; and if an how consumers were involved in the project; (18) the publication status of the project; and whether they believed the results were useful.We also asked respondents about their overall experience, including satisfaction with the overall experience, skill development opportunities, and research engagement after training.
To gain a deeper understanding of their experiences we asked trainees to complete two additional validated questionnaires.The Postgraduate Research Experience Questionnaire (PREQ) is .traditionally used to assess experiences of higher degrees by research graduates across 7 domains: supervision, intellectual climate, skills development, infrastructure, thesis examination, goals and expectations, and industry engagement.(19) We excluded the industry engagement domain as our participants were based in industry.The WReN Spider instrument (20) assessed trainees' selfperceived end-of-training research knowledge, skills and experience, focusing on the individual's experience rather than the broader research environment.(21) Quality assessment of research outputs.
We were unable to source trainee outputs directly from colleges, as some colleges do not archive submitted reports and others require trainee consent to release their reports.Therefore, we asked participants to upload a copy of the manuscript that was submitted to the college to the survey, or a citation of the published work.We assessed whether there was a clear question, a study rationale, adequately consideration of the published literature, and a sample size calculation (where relevant).Depending on their study type, we appraised the quality of reporting using EQUATOR guidelines (22) and the study methods using relevant critical appraisal tools (Supplementary File 2).

Sample size.
As we did not test a hypothesis, we did not undertake a formal sample size calculation taking a more pragmatic approach.(23) Using an acceptable margin of precision of ± 10% for standard prevalence estimates, and a worst-case rate of completed and uploaded research projects of 20%, (24) we estimated a sample size of around 480 responses to yield 96 completed research projects for analysis.

Analysis.
We included responses in the analyses if participants completed at least one demographic data question and analysed the survey using simple descriptive statistics.We did not conduct any sensitivity analysis or use any methods for adjusting for potential non-representativeness of our sample.Due to small response rates of trainees from individual colleges, differences in trainee responses between colleges were not explored, and factor analysis of validated surveys was not conducted.Data analysis and visualisation were conducted using Python 3.10.9(25)and R 3.6.1 (26) using Jupyter notebooks (27) on a Windows 10 64-bit operating system.
Open-ended survey responses were analysed using qualitative content analysis where core meaning is derived from the text and grouped into themes.(28) This was conducted by an .experienced qualitative research assistant in Microsoft Word.Themes were discussed by a subset of the team with content and qualitative research expertise (PS, CB).

Ethics
This study was approved by the Bond University Human Research Ethics Committee (PS00149).

Results.
Of the 426 eligible participants who commenced the survey, 371 (87%) completed at least one demographic question (Figure 1), with the median time for survey completion being 5.3 minutes.

Demographic data.
Approximately two thirds of respondents were female and currently undergoing training.Of those who had completed their training, over half (58%) had finished in the previous two years.Most were undertaking their training in urban centres (Table 1).Participants represented all but one of the 16 medical specialty colleges in Australia and New Zealand (Table 1), with 323 respondents from Australia and 48 from New Zealand.Respondents from Queensland, the investigators' home state, were over-represented (Supplementary Table 1).

Research Projects.
Of the 361 respondents who provided information, 311 (86%) had completed, were completing, or were planning to complete a project (Table 2).Of 47 who provided reasons for not conducting projects, 20 reported that it was not required by their college, 13 completed approved coursework and none had completed a PhD (Table 2).A third of respondents completed more than one project (48/174, 27%), equating to a total of 267 projects conducted by 173 trainees.

Study provenance.
Of 177 trainees whose projects were completed, only 79 (45%) responded providing further information on 92 projects.Thirty-eight (41%) projects addressed questions developed by the trainees alone (Table 3).Some questions arose during clinical discussion (35/92, 38%) and a few were part of ongoing research (14/92, 15%) (Table 3).Forty-eight (52%) of study designs were developed by the trainees with little input from others.Of the 69 developed protocols, 60 were developed by the trainee and 20 were registered, including 11 in journals and 7 in registries.Most trainees searched for a systematic review of the literature prior to starting their project (68/92, 74%). .

Project support and collaboration.
Trainees reported low levels of interdisciplinary and interprofessional collaboration, with 43% (39/90) of project teams consisting of only members of their own specialty.Forty-four percent (40/90) of project teams consisted of only one other profession, often a statistician, allied health professional, or nurse, but seldom other disciplines (Table 3).Most respondents (68/85, 80%) reported obtaining some expert support -most commonly clinical expertise, library services and study design or measurement expertise (Table 3).Only 7 out of 90 (8%) projects involved consumers (Supplementary Table 2).Most (57/85, 67%) reported that they received adequate support from their supervisor.Trainees from all but two colleges reported carrying out the research work in their own time (Table 4).The PREQ survey explored research support further.Only 10 responded (Supplementary Figure 1).Respondents were least satisfied with the intellectual climate in which they conducted research and opportunities for skills development, and most satisfied with the thesis examination process and understanding project requirement goals and expectations.

Perceived value of the research findings and dissemination of results
Most participants (78/92, 88%) indicated the results of their study would be useful in practice (78/92, 88%), and the majority (81/90, 90%) felt confident using the results in practice.(Table 5).
Half of the projects (46/90, 51%) have a publicly available manuscript.Of those published in a journal, the trainee was usually first author (37/42, 88%) and approximately a third (33/90, 37%) were published by the end of training.

Respondents' views on mandatory research projects.
Almost half the participants who completed a project (87/174, 50%) felt that this effort was very or moderately important to their clinical career and over half (40/68, 59%) felt that completing a research project improved their ability to read and interpret research (Table 5).
The participants responses on the value of conducting mandatory research projects were mixed, with around half of the respondents expressing positive attitudes (Supplementary Table 3).When asked why, 236 participants provided a response.Sixty-five (27%) participants mentioned the time required to do the research was unreasonable given clinical workloads and time away from family life and other priorities (Table 6).Fifty-one participants (21%) felt mandatory projects contributed to poor quality research and 21 (9%) described them as "tick box" activities.Thirty-nine (17%) participants described a lack of structured support in the current training program, 36 (15%) suggested the research projects were a waste of time or not relevant to their career objectives, and 28 (12%) suggested there were better ways to learn EBP or research skills.While 29 (12%) . recommended research should be optional rather than mandated, 44 (19%) participants suggested mandatory projects were important to develop skills beyond just research, 18 (8%) suggested it improved their EBP skills and 14 (6%) suggested they improved clinical practice.
Self-perceived end-of-training research skills were explored further using the WReN Spider instrument.Only 10 responded (Supplementary Figure 2).All respondents felt they were somewhat to very experienced in finding and appraising the literature, and less than half felt this way about protocol writing, publishing, qualitative methods and acquiring funding.
Since completing their training, almost two thirds of respondents (44/61, 72%) had thought about initiating new research after completing their training, and approximately half had participated in research (Table 5).When asked the reason for their answers in free text, 36/56 (64%) participants commented they now had more time and interest to participate in research, whereas 21/56 (38%) said they had no time (Table 6).
Research outputs -quality of methods and reporting.
Respondents uploaded 34 studies (Supplementary Table 4); 28 were assessed for quality as six were excluded due to a lack of standardised instruments.
Overall, the introduction and discussion sections were well reported; however, there were gaps in reporting in other sections (Figure 2).Most studies had moderate to high risk of bias; 3/27 were deemed to have low risk of bias and one was unassessable (Supplementary Figure 3).

Discussion.
Most trainees in this study were required to complete a research project as part of the specialty training.Overall, responses indicated that educational experiences and quality of research outputs were inconsistent.However, in our view the most significant finding of our study was the significant attrition of responses at each stage of the survey.Of 177 trainees who reported completion of a research project, just over one third of trainees responded to questions regarding how they conducted their project, and one sixth uploaded a project.We feel that those who had a better experience were more likely to respond providing a more positive picture than might be observed more broadly.Despite this, our results are enlightening.
Approximately half of the respondents were solely responsible for developing their research questions, designing their studies, and developing study protocols, while only few projects were part of ongoing research.Although most reported adequate support from their project supervisors, .they worked in non-collaborative teams often with only their own specialty.Although statisticians, allied health or nurses were frequently represented in research teams, trainees reported low levels of access to additional expertise.This may reflect lack of research opportunities and resources or the view that medical specialists are expected to learn by doing and leading research irrespective of their baseline abilities.
Most conducted projects in their own time.Exceptions to this were those from the Royal Australian College of General Practitioners, which provide protected time for 20 trainees per year and don't require a project from others, and the Royal College of Pathologists of Australia, which conducted most of their research during clinical time.
When we assessed uploaded projects' design and reporting quality, reporting standards were met to a fair degree, and few reports had study methods judged to have a low risk of bias.
The trainees who provided details of their completed projects reported some positive features.
Most searched the literature for systematic reviews before starting, two thirds drafted research protocols and over one third were registered.Half of the project reports were published in journals, usually prior to respondents completing their training.Most thought their project's findings were useful and likely to use them in clinical practice, and that the experience of conducting the research project was important to their career.
Negative views were expressed by those both supportive and unsupportive of mandatory projects including conducting projects uncompensated in their own time, competing with family commitments; a lack of structured support; and concerns their projects were 'tick box' projects which simply contributed to unhelpful research findings -in other words, research waste.
Respondents also commented that learning how to apply research evidence in practice as preferable to conducting projects.
Our study has limitations.While we are unable to judge the true representativeness of this sample, our results are likely biased towards more positive experiences.The uploaded project reports had a higher publication rate compared to the broader study cohort and in health and medical literature more generally (74% v 50%) (29-31), likely representing better quality studies than the broader trainee population.Since completing their training, over 70% of respondents had considered participating in research and over half had become involved in research, figures much higher than previously reported for Australian specialist medical practitioners.(32) This suggests our .respondents may have a higher interest in research than the broader trainee population.There is probably a large silent majority who withheld generally negative views.
The real value of this educational approach can be judged by asking the right questions: 'What do we aim to achieve by providing research training to doctors?And what is the best way to get there?'While government reports suggest the need for better translation of research into practice and familiarity with contemporary research methods, (33) clear strategies for achieving these goals have not been clearly enunciated by the relevant professional bodies.
Most will agree that every practitioner should be competent in translating research findings into their practice; however, requiring every trainee to undertake a research project to teach EBP is not fit for purpose.(34)Some, like the Royal Australian College of General Practitioners and Australasian College of Emergency Medicine, have recognised this.(12,35) At present less than 1% of Australian doctors identify as being a researcher, and less than 8% participate in research.(32,36).The small number of trainees who go on to be research leaders will be internally driven to do so and should be well supported from an early stage.This leaves a substantial number who could contribute to worthwhile collaborative research enterprises (e.g., participation in large adaptive platform trials and observational studies) but who are not currently being prepared for this activity.
Trainee research collaboratives (TRC) are a potential avenue to learn such skills.These have been used in the United Kingdom since 2007 and produce high quality research while providing developing trainees skills.(37,38) TRCs are beginning to form across Australia and New Zealand, (39) but unless trainees are first authors, contributions rarely receive college recognition.
We believe there are two important unintended consequences of this well-meaning tradition of leading research for specialist qualification.First is the likely contribution to the wider issue of research waste though poorly planned and executed projects.However, being able to support every trainee to lead a study that meaningfully contributes to the scientific body of literature takes substantial resourcing that is neither feasible nor sustainable.The second, and perhaps more significant implication, is the missed opportunity when trainees are tasked with leading research instead of honing research skills relevant to their career objective -which, for most trainees is to be an evidence-based clinicians -and to prepare clinicians for collaborative research.Future work should articulate a minimum set of research competencies and develop a flexible training curriculum that can be adapted to the career needs and aspirations of individuals.

"I think in general mandatory research requirement to produce "papers" contribute to a large bubble of generally irrelevant papers which adds to a constant background of research noise that doesn't actually change practice." (Moderately opposes mandatory projects)
"There is already an abundance of very questionable registrar level "research" diluting the pool of genuine, high quality, and clinically useful publications that are available.Cynically completing a research project because you are force [d] to do so does not benefit the individual or the profession, rather the opposite.(  .Table : Reporting and design quality assessment tools.* We had 1 animal study which was a randomised trial and could therefore be evaluated using the MMAT quantitative randomized controlled trial tool.†The AMSTAR signalling questions were modified slightly to allow for evaluation of non-interventional studies and scoping reviews.This was done with 3 team members, 1 of which is an author on the original AMSTAR tool (DH), a statistician (AB), and systematic review expert (ABB).‡ Risk of bias elements only.

TOOL USED STUDY DESIGNS Reporting Quality AGREE(3)
Guidelines ARRIVE (4) Animal studies CARE (5) Case reports, Case Series CONSORT Crossover (6) Cross over randomised trial PRISMA (7) Systematic Reviews PRISMA-ScR (8) Scoping Reviews SRQR (9) Qualitative STARD (10) Diagnostic test accuracy STROBE (11) Cohort Eligible participants were those completing, or who had recently completed (within the past 5 years) a specialty training program at an accredited Australian or New Zealand specialty training college.No other inclusion or exclusion criteria were used.

Figure 2 :
Figure 2: Quality of ReportingPanel A contains 27/28 evaluated manuscripts.Each line represents an uploaded manuscript.One manuscript was a clinical guideline and did not map to same sections (i.e.: title, abstract, introduction, methods, results, discussion and other) and can be found in Panel B. Some lines are blank because they were an abstract only (Article 20) or because CARE reporting guidelines do not have a methods section (Articles 15,26,35,38) and the ARRIVE reporting guideline (Article 37) merges title and abstract together.

Table 2 :
Demographic data. of those eligible to answer the question.Abbreviations: RACP: Royal Australasian College of Physicians, ANZCA: Australian and New Zealand College Of Anaesthetists, RANZCP: Royal Australian and New Zealand College of Psychiatrists, RACGP: Royal Australian College of General Practitioners, ACEM: Australasian College for Emergency Medicine, CICM: College of Intensive Care Medicine, RACS: Royal Australasian College of Surgeons, RANZCOG: Royal Australian and New Zealand College of Obstetricians and Gynaecologists, RCPA: Royal College of Pathologists of Australasia, RANZCO: Royal Australian and New Zealand College of Ophthalmologists Project completions and reasons for non-completions * of those eligible to answer the question.

Table 3 :
Project conductSurvey questions relevant to understanding how respondents conducted their research.* of those eligible to answer the question.†Project numbers add up to more than those that answered as each project could have answered "yes" to more than one category.

supervisor provided you with adequate research support while conducting your scholarly project.
.

Table 4 :
When projects were conducted.asked participants to estimate the percentage of time they spent on their scholarly projects during scheduled service/clinical time, protected time or during their own time.We received responses for 85/267 projects.Red highlights indicate highest median number for that row.RACGP's trainee program offers an academic post which provides funding for protected research time.* 5 participants gave information on 10 projects, † 1 participants gave information on 1 project, ‡2 participants gave information on 4 projects, §1 participant gave information on 7 projects.Abbreviations: ACEM: Australasian College for Emergency Medicine, ACSEP: Australasian College of Sport and Exercise Physicians, ANZCA: Australian and New Zealand College Of Anaesthetists, CICM: College of Intensive Care Medicine, RACDS: Royal Australasian College of Dental Surgeons, RACGP: Royal Australian College of General Practitioners, RACMA: Royal Australasian College of Medical Administrators, RACP: Royal Australasian College of Physicians, RACS: Royal Australasian College of Surgeons, RANZCOG: Royal Australian and New Zealand College of Obstetricians and Gynaecologists, RANZCP: Royal Australian and New Zealand College of Psychiatrists, RANZCR: Royal Australian and New Zealand College of Radiologists, RCPA: Royal College of Pathologists of Australasia

Table 5 :
Project value* of those eligible to answer the question.

gaining your most recent fellowship, have you participated in any research projects as an investigator?
.

Table 6 :
Codes from free-text content analysis

Reasons trainees conducted research after their training:
"Almost the entire project is done in my spare time, this ended up being hundreds of hours... there is no access to any kind of research resources by the college, other than a handful of PDFs of previous .projects on the website.It's a great idea, but as a trainee, I am tired of being forced to spend my spare time outside of work (when I should be relaxing/having a family/doing hobbies) devoted to mandatory training that is not supported by the college.We are stuck doing boring projects… on our own time, and end up with the worst of both worlds."(Strongly supports mandatory projects) "Not enough support, guidance or time provided for project work.It is extremely difficult to find time to complete your project as well as an appropriate supervisor with the time, interest and experience in research."(Strongly opposes mandatory projects) "I can see some virtue to this, but the implication that every specialist has to be a researcher is invalid.Additionally, the requirement to conduct your own study and be first author (as opposed to participating in a multi-centre study) excludes a lot of "It is valuable to participate in research though and to learn the finer points and have better understanding of the process.It would perhaps be more valuable to assess the quality of the project participated in and the contribution rather than the first author status."(Moderately opposes mandatory projects) "The skill in interpreting research is much better taught in an academic environment rather than forcing people without any background in research to complete often low-quality research in an unsupported manner.(Strongly opposes mandatory projects) . . .

country and state did you complete/ are completing most of your most recent specialty training in?
, Case control, Cross Sectional Design Quality MMAT(12) Animal studies*, Case reports, Case Series, Cross over randomised trial, Qualitative, Cohort, Case control, Cross Sectional . Supplementary

Table 2 :
Additional information on study conduct * Number of participants.† of those eligible to answer the question.

REPORTED BY TRAINEES. I had access to a good research-related seminar(s) or training program.
.