Reporting of patient involvement: A mixed-methods analysis of current practice in health research publications

Objectives To evaluate the extent and quality of patient involvement reporting in examples of current practice in health research. Design Mixed-methods study. We used a targeted search strategy across three cohorts to identify health research publications that reported patient involvement: publications published in The BMJ, publications listed in the PCORI database, and publications citing the GRIPP2 reporting checklist for patient involvement or a critical appraisal guideline for user involvement. Publications were coded according to three coding schemes: "Phase of involvement", the GRIPP2-SF reporting checklist, and the critical appraisal guideline. Outcome measures The phase of the study in which patients were actively involved. For the BMJ sample, the proportion of publications that reported patient involvement. The quality of reporting based on the GRIPP2 short form reporting guideline. The quality of patient involvement based on the critical appraisal guideline. Quantitative and qualitative results are reported. Results We included 87 publications that reported patient involvement. Patients were most frequently involved in study design (90% of publications, n=78), followed by study conduct (70%, n=61), and dissemination (40%, n=35). Reporting of patient involvement was often incomplete, e.g., only 39% of publications (n=34) reported the aim of patient involvement. While the methods (56%, n=49) and results (59%, n=51) of involvement were reported more frequently, qualitative analyses showed that reporting was often unspecific and the influence of patients' input remained vague. Therefore, a systematic assessment of the quality and impact of patient involvement according to the critical appraisal guideline was not feasible across samples. Conclusions As patient involvement is increasingly seen as an integral part of the research process and requested by funding bodies, it is essential that researchers receive specific guidance on how to report patient involvement activities. Complete reporting builds the foundation for assessing the quality of patient involvement and its impact on research.

and dissemination (40%, n=35). Reporting of patient involvement was often incomplete, e.g., only 39% of publications (n=34) reported the aim of patient involvement. While the methods (56%, n=49) and results (59%, n=51) of involvement were reported more frequently, qualitative analyses showed that reporting was often unspecific and the influence of patients' input remained vague. Therefore, a systematic assessment of the quality and impact of patient involvement according to the critical appraisal guideline was not feasible across samples.

Conclusions
As patient involvement is increasingly seen as an integral part of the research process and requested by funding bodies, it is essential that researchers receive specific guidance on how to report patient involvement activities. Complete reporting builds the foundation for assessing the quality of patient involvement and its impact on research.

PROTOCOL
The protocol was published on the Open Science Framework: https://osf.io/vntgu/

STRENGTHS AND LIMITATIONS
 A targeted search strategy was used to identify examples of patient involvement reporting in a variety of publication types and study designs in health research  A mixed-methods approach allowed for an analysis of both the completeness and quality of patient involvement reporting  In this study, we coded statements reporting on patient involvement in 87 health research publications that may be adapted for further use  Reporting of patient involvement was insufficiently detailed to allow for a systematic assessment of the quality of patient involvement

INTRODUCTION
Patients' viewpoints should be included in clinical research as they are the most affected by it. [1] Different approaches can be used to make the outcomes of clinical research more relevant to patients. One option is to actively involve patients or patient representatives in study design, study conduct, and dissemination. Different terms are used to describe this active involvement, e.g., "Patient and Public Involvement" (PPI) or "Patient Engagement". Patient involvement in health research varies widely and can be categorized for example according to the level or continuity of involvement, involvement in different phases of the research, or the methods applied for involvement. [1,2] Standards and principles for patient involvement focus mainly on the management of the relationship between patients and researchers. [3] These principles are important for a good collaboration, but only reflect one quality aspect of patient involvement or its impact on outcomes of clinical research. Given the efforts from patients, there is also an ethical imperative to reflect about their input in the publication, and the results and the impact of patient involvement should be evaluated and published.
Quality in clinical research is assessed with critical appraisal tools, such as the widely used risk of bias tool for randomized controlled trials (RCTs). [4,5] A critical appraisal tool to assess the quality of patient involvement was developed in 2010. [6] High quality reporting is needed to allow for critical appraisal and quality assessment. A reporting guideline GRIPP (Guidance for Reporting Involvement of Patients and Public) was developed in 2011 [7] and updated in 2017 to GRIPP2. [8,9] GRIPP2 comes in two different formats: a long form (LF) for studies with patient involvement as primary focus and a short form (SF) for studies with patient involvement as secondary or tertiary focus, such as for example clinical studies being informed by an active involvement of patients.
In 2014, the British Medical Journal (BMJ) group endorsed a policy which made it a requirement to report on patient and public involvement in BMJ journals and recommends GRIPP2 as a reporting standard. [10] Price et al. [11] compared reporting of patient and public involvement (PPI) before and after the introduction of the BMJ policy. They found that while 86% of research articles included a PPI statement about one year after the introduction of the policy, only 11% actually reported PPI activities. Funding organizations are also likely to play an important role in improving the quality and reporting of patient involvement, especially as they increasingly require patient involvement in clinical research. [12] The objective of this study was to analyse the extent and quality of patient involvement reporting and the quality of patient involvement in examples of current practice in health research. We are aware of several studies that investigated the rate and quality of reporting [13] or critically appraised patient involvement in specific domains of clinical research. [12,[14][15][16] Overall, these previous assessments identified very few publications that reported patient . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted April 21, 2022. ; https://doi.org/10.1101/2022.04.20.22274033 doi: medRxiv preprint involvement and/or engagement and reporting quality was sub-optimal (e.g., 0.4% of the sample described the active involvement of patients in orthopaedic research [16]). Jones et al. [15] also included studies which had patient involvement as primary focus, such as prioritizing research topics. Our scope was different: We focused on studies that actively involved patients to inform the study methodology (including dissemination) but did not have patient involvement as primary focus. We did not limit our analysis to a specific research area or experimental design (e.g., RCTs), but included three purposively selected cohorts of publications in which we expected reporting of patient involvement. We considered patients as people affected by the disease or topic, their family members, or representatives of those affected.

METHODS
A protocol detailing the methods of this study was pre-registered in the Open Science Framework (April 28, 2020) and is openly available.[17, see also Supplement 1]

Samples
In order to identify publications that report on patient involvement, we used a targeted search strategy in the following three samples: (1) Publications in the journal The BMJ, which requires reporting on patient involvement in research articles. We performed a Web of Science search (Web of Science Core Collection, March 4, 2020) to obtain all publications published in 2019 in The BMJ (document types: 'Article' or 'Review').
(2) Publications listed in the PCORI database. [18] PCORI is a US-based organization funding patient-centred research, which continuously screens Medline via PubMed, relevant journals, and PCORI staff recommendations for publications on patient engagement in health research.
We filtered for topic: example of engagement in health research; stakeholder involvement: patients; year: 2019.
(3) Publications citing one of the two GRIPP2 publications [8,9] or the critical appraisal publication [6] in Dimensions. [19] Details of the search strategies can be found in Supplement 2. All included publications across these samples were checked for additional links or references, which described patient involvement in more detail. If additional relevant documents were found (e.g., supplemental materials, previously published protocols), they were included in the sample.

In-and exclusion criteria
The following inclusion and exclusion criteria were applied across all samples: . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)  Systematic reviews and scoping reviews were included; narrative reviews were excluded.
 Mixed-method studies and those which used qualitative and quantitative methods were included if the methods were mainly quantitative.  Protocols were excluded. If an original publication in our sample cited a protocol which provided more detailed information on patient involvement, this protocol was included as an additional document.
 Studies in which a tool was developed and tested were included. Studies in which tools/interventions/outcomes were developed but not applied were excluded.  Comments, editorials, guidelines, consensus papers, and other publications, which did not aim to answer a research question, were excluded.

Patient involvement:
 Publications were included if a patient involvement activity was described in at least one phase of the study, i.e., patients or patient representatives were actively involved in designing or running the study, were engaged as co-researchers, supported the dissemination of results, or had an advisory function; participating in a study as a "subject" or "participant" was not considered as sufficient to qualify as patient involvement.
 Publications were included if patient involvement was used to inform the study, but it was not the primary focus of the study. The authors' decision whether to complete the short or long form of the GRIPP2 reporting checklist was used as an indicator of the focus of the study (if applicable).  Publications were included if patient involvement activities had already been conducted (not only planned). The only exception was for patient involvement in dissemination activities. Given that dissemination activities often take place after a study is published, studies with planned patient involvement in dissemination activities were included.
All identified studies were screened by at least two members of the team. Discrepancies were discussed in the team until consensus was reached. Inclusion and exclusion criteria were refined during the process to accommodate for the wide variety of studies included in the . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted April 21, 2022. ; https://doi.org/10.1101/2022.04.20.22274033 doi: medRxiv preprint sample. Deviating from the protocol, we decided to exclude all qualitative studies and mixedmethods studies with mainly qualitative methods. In most of these publications, it was difficult to distinguish between the active involvement of patients in the study and their involvement as subjects of the qualitative research.

Coding
We coded all publications and additional documents using three coding schemes (Supplement 3): 1. Phase of involvement: Included publications had to report patient involvement in at least one of the three study phases: study design (sub-codes "research question" and "outcome measures"), study conduct, or dissemination (sub-code "co-authoring the manuscript").
3. Critical appraisal tool [6] to assess the quality of patient involvement.
We used an inclusive and pragmatic approach in the coding. For example, we accepted a statement such as "patients were included to inform the study design" as sufficient to describe the aim of patient involvement according to GRIPP2. A mere description of tasks was considered as sufficient to code "have the researchers discussed the nature of tasks" according to the critical appraisal tool. We also coded acknowledgement and contribution statements if these mentioned phases or activities of patient involvement. In view of assessing the quality and impact of patient involvement based on included publications, we coded statements that addressed criteria in the critical appraisal tool. However, the varying amount of detail reported across publications and samples did not allow for a systematic appraisal of the quality of patient involvement.
Two raters coded all included publications according to the coding schemes. Discrepancies were discussed until consensus was reached. If this was not possible, a third person assessed the respective passage and the team decided by majority vote.

Analysis
Coded segments were exported from MAXQDA [20] and analysed further in Microsoft Excel.
Codes from additional documents were merged with that of the original study. We quantified how many publications in each sample reported one of the codes at least once. For the BMJ sample we additionally report the frequency of patient involvement across all publications, given the journal's requirement to report whether patient and public involvement has taken place. [10] . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted April 21, 2022. ; https://doi.org/10.1101/2022.04.20.22274033 doi: medRxiv preprint Additionally, we conducted a qualitative content analysis based on the extracted GRIPP2 codes. Similar codes within the same GRIPP2 category were grouped into overarching themes. All coded segments were also reviewed for illustrative examples.

Patient and Public Involvement
Patients or the public were not involved in the planning or conduct of this meta-research study.
The analyses were not restricted to studies on specific diseases or patient populations; therefore, it would not have been adequate to include a specific patient group since this research is not specifically relevant for them. The main target audience includes researchers and other stakeholders in health research (e.g., journal editors, funders). The results have been discussed in workshops with health researchers and patients and/or patient representatives and other stakeholder (e.g., funders) to raise awareness of this topic and to describe the progress of integrating patient involvement in health research.

In-and Exclusion
A total of 87 research publications were included in the analysis after applying our inclusion and exclusion criteria (see Supplement 2). From the BMJ sample, 32 of 155 research articles (21%) were included because they reported PPI activities and qualified as quantitative study.
We included a further 42 publications from the PCORI sample and 13 from the Citation sample (12 citing GRIPP2 and one citing the critical appraisal tool). Most frequently applied exclusion criteria were "no patient involvement" and "no research publication". We included 35 additional documents, which provided further information on patient involvement described in the publications. [21]
In 17% (n=15) of the publications, patients were involved in formulating the research question and in 31% (n=27) in defining outcome measures.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

GRIPP2 (short form)
Between 13% (n=4, BMJ sample) and 77% (n=10, Citation sample) of the publications reported the aim of patient involvement (  E1: vague, unspecific statement "He was recruited to provide a patient's perspective" [23] E2: elaborated, informative statement "To ensure an improved completion rate in the full study, we hired a patient partner (B.G.S.) to approach bedside nurses, enrol participants and assist family caregivers with the completion of the questionnaires. We expect this will increase the completion rate because this family caregiver has the lived experience, perseverance and communication skills that are necessary to make the consent process less overwhelming." [24] Methods Provide a clear description of the methods used for PPI in the study E3: broad statement lacking details "Patient discussion groups were used as a means of involving patients in setting the research question and for determining the outcome measures." and "The outcome measures were identified after discussion groups in our unit, with patients who had previously undergone knee arthroplasty surgery." [25] E4: detailed, informative statement "During the pre-award period, the researchers developed a stakeholder management plan ("roadmap") outlining where stakeholders could influence the study. (…) Creating a stakeholder roadmap allowed us to target defined actions to engage stakeholders quickly, even before the study begun. Our stakeholder engagement plan focused on 4 key sets of activities: (1) study planning (including study design, intervention design and procedures, outcomes measurement, materials); (2) hospital/patient recruitment and retention; (3) study implementation; and (4) translation, including interpreting study findings and disseminating results back to participating communities and the public." [26] Study results Report the results of PPI in the study, including both positive and negative outcomes 28% (n=9) 76% (n=32) 77% (n=10) 59% (n=51) E5: broad statement lacking details "Their input helped to refine the research question and to improve the protocol considerably." [27] E6: informative statement "The group stated that participant-facing material (participant information leaflet and invitation brochure) were not inclusive enough for all 'father figures' (e.g., stepfathers). The material in question was changed to be more inclusive. To boost recruitment to the trial, the PPI group suggested organisations that might be accessed by our target population and that might be interested in supporting the study (e.g., youth clubs, after-school clubs and sports centres). Therefore, we approached and successfully recruited fathers from after-school and martial arts clubs and Scouts groups for the cultural adaptation phase. When asked about dissemination of study results, PPI representatives thought that it was important to report the findings back to the local authorities that funded the HDHK programmes and to include the use of the local authorities' social media channels for dissemination. The group encouraged the research team to be fully open about the challenges faced in delivering the HDHK programme and the research study." [28] Discussion and conclusions Comment on the extent to which PPI influenced the study overall. Describe positive and negative effects.
E7: statement reporting on influence on usability "We enhanced the usability of our findings by engaging key stakeholders (i.e. clinicians, parents and researchers) as part of our integrated knowledge translation activities (..). Their guidance ensured that the review findings were both clinically meaningful and family-centred." [29] E8: statement providing details on the influence on the primary outcome "Patient prioritization of meaningful physician discussion over reducing anxiety and depression, therefore, resulted in a substantial redesign of both the ACP video and its evaluation trial. The team negotiated with its funder to change the primary outcome of the video's evaluation from anxiety and depression to having a meaningful discussion. Meaningfulness of the discussion was operationalized by the measure 'patient-centered nature of the patient-surgeon conversation' in the resulting video evaluation trial, procedures for which have been previously . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted April 21, 2022. ; https://doi.org/10.1101/2022.04.20.22274033 doi: medRxiv preprint described. This change in outcome also required that language be added to the video to encourage open communication between the patient and surgeon. In the practice-oriented context of the current study, using both engagement and research approaches in endpoint selection provided an innovative means of identifying and prioritizing endpoints." [30] Reflections/critical perspective Comment critically on the study, reflecting on the things that went well and those that did not, so others can learn from this experience E9: statement reflecting on organizational issues "Originally, we intended to conduct these sessions in a group format, due to difficulties with PPI partners' schedule commitments, one-to-one sessions were conducted." and "It is extremely important that researchers plan PPI at the grant proposal stage and estimate the costs appropriately. If these costs are not correctly estimated during the initial stages of developing research proposals, they may cause a financial burden on PPI partners." [31] E10: statement reflecting on representativeness "One limitation will be engaging patients and family members who are deeply committed to the project and who consequently may not provide a representative range of patient-family perspectives. However, their insights will be vital to identifying key aspects of patient/family-centered decision aids and they will directly inform the next stage of our research-cognitive interviews with patients and their families about advance care planning aids." [32] Critical appraisal tool The critical appraisal tool [6] focuses on the quality and impact of user involvement in research (Table 3; [21]). Specific appraisal criteria were rarely reported, such as discussing the level of involvement (7%, n=6), considering whether findings were disseminated appropriately to recipients (7%, n=6), or conducting a formal evaluation (6%, n=5). More general appraisal criteria were reported more frequently, such as the nature of tasks patients were asked to perform (45%, n=39), how findings were disseminated (not requiring an active part of patients) (46%, n=40), or a general evaluation of the added value of involving patients in the research process (47%, n=41).
Further appraisal criteria that were addressed in very few publications were the nature of training of patients (8%, n=7) and researchers (1%, n=1), or ethical (3%, n=3) or methodological (6%, n=5) considerations and how these were managed (3%, n=3 and 5%, n=4, respectively). . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted April 21, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted April 21, 2022.

DISCUSSION
We analysed a sample of 87 publications in health research that reported on patient involvement. While many publications provided information on general aspects relating to patient involvement, even very basic details were often lacking. For example, nearly all publications reported generally that patient involvement took place during study design.
However, more specific information about whether this involvement in study design included defining the research question or prioritizing outcome measures was reported to a much lower extent. Similarly, 39% and 56% of publications reported on GRIPP2 aims and methods, respectively, but the reporting was often sub-optimal and statements rather vague. Despite authors alluding to many aspects of patient involvement included in the GRIPP2-SF and critical appraisal guidelines, we identified a need to improve completeness and details of reporting.
This corroborates findings from previous studies on reporting of patient involvement and/or engagement. [12,13,16] Due to this incomplete reporting, coding according to the GRIPP2-SF categories was a challenge: the sparseness of reporting in many of the publications meant that these categories were relatively broad and overlapping. For example, a statement such as "patients helped with the identification of meaningful outcomes" could describe the method (i.e., focusing on the process) or the results (i.e., identifying outcomes) of patient involvement. While the GRIPP2-SF reporting checklist is certainly useful to guide reporting in studies not having patient involvement as primary focus, our findings suggest that complementary measures could further bolster its impact on the quality and consistency of the patient involvement evidence base. Such measures could for example include broader requirements to include a statement on patient involvement in publications, more specific guidance for authors and peer-reviewers, and standardised formats without word count restrictions to support more complete and consistent reporting. High quality reporting is the basis for assessing the quality of patient involvement.
We observed considerable differences between our three samples regarding reporting, with both the Citation and PCORI sample providing more information on patient involvement than . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted April 21, 2022. ; https://doi.org/10.1101/2022.04.20.22274033 doi: medRxiv preprint the BMJ sample. This is not surprising given the expected emphasis on patient involvement in the former samples compared to providing information in a mandatory section.
For the BMJ sample, we assessed the percentage of publications that reported patient involvement in the mandatory PPI section. Of all research articles published in 2019 (n = 155), 21% reported patient involvement activities. In the sample of Price et al. [11], which included research articles published between June 2015 and May 2016, patient involvement was reported in only 11% of the articles. Thus, the proportion of research articles reporting on patient involvement doubled in only a few years, demonstrating the impact of this journal policy to enhance visibility and to raise awareness for patient involvement. While this trend is encouraging, descriptions of patient involvement in the BMJ sample were generally very short and did not elaborate on the results of patient involvement or provide a thorough description of the process. For example, several publications reported on the inclusion of patient discussion groups without describing the composition, specific tasks, or influence of this group.
In contrast, in the PCORI and Citation samples we often found very detailed descriptions of patient involvement, including the nature of performed tasks, concrete examples of its influence on the study, and critical reflections [see 21 and table 2]. In many cases, these descriptions were provided in additional documents. This suggests that the word count limit imposed by journals likely contributes to the limited detail in patient involvement reporting.
Additional documents or structured tables for reporting of patient involvement may be helpful.
However, this approach may come with the risk that patient involvement is seen as an add-on rather than as an integral part of the conducted research.

Strengths and limitations
One of the strengths of our study was the use of a targeted search strategy to identify examples of patient reporting in current practice across a variety of publication types and study designs in health research. Moreover, coding according to three distinct schemes allowed us to capture different aspects of relevance, including the phase of patient involvement, the use of and adherence to reporting guidelines (GRIPP2-SF), and the quality and impact of patient involvement (critical appraisal tool). All statements reporting patient involvement analysed in this study are openly available [21] for further use. For example, coded statements may inform the development of automated tools to detect reporting of patient involvement in publications.
We could not systematically assess the quality of patient involvement according to the critical appraisal criteria as originally planned. Quality assessment highly depends on reporting completeness and detail, which was inconsistent across publications. Such an analysis in our diverse sample might have favoured long vs. short reports, or participatory health research approaches vs. PPI activities informing a clinical trial.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted April 21, 2022. ; https://doi.org/10.1101/2022.04.20.22274033 doi: medRxiv preprint Initially, we did not plan to exclude publications applying qualitative research methods.
However, we did not find a clear definition to differentiate between the active involvement of stakeholders and their involvement as participants in qualitative research, for example in focus groups or interview studies.[e.g., [33][34][35] In some cases, both were reported in the same publication.[e.g., 33] This particular challenge has previously been noted in the context of assessing reporting of patient and public involvement. [11] Despite attempts to delineate these approaches [36], this is not common practice yet. More generally, excluded qualitative research studies that reported patient involvement often had patient involvement as primary focus. An analysis of these studies was beyond our scope.
We used GRIPP2 as a reporting guideline to assess the completeness of reporting of the included publications. The use of reporting guidelines without modification to serve as evaluation tools has been questioned by Logullo et al. [37] as their purpose is to guide writing.
However, the authors of GRIPP2 explicitly stated that it can also be used for planning patient involvement or for quality assurance. [8] Conclusion Despite important developments in the last years, patient involvement is still not a wellestablished approach in clinical or health research. [14,38] Therefore, we would encourage journals to request an obligatory patient involvement statement from their authors, and to give guidance on detailed reporting in a structured table or additional document. We would also encourage journals and funding organizations to support the reporting of patient involvement by requiring the use of GRIPP2-SF as a reporting tool. Finally, we encourage researchers to include sufficient detail on patient involvement in their study to allow others to derive and apply lessons learned in their own studies.
We expect that patient involvement will become more important in the next years to increase the relevance of research, in line with increasing demand from funders, publishers, and society.
Broader implementation of policies and more specific guidance are needed to leverage the impact of existing reporting guidelines, and thereby improve the quality of the patient involvement evidence base. Complete reporting builds the foundation of assessing the quality and appropriateness of patient involvement and is essential towards increasing its impact on research.