Validation of questionnaires and rating scales used in medicine: Protocol for a systematic review of burnout self-reported measures

Rationale

In the era of evidence-based medicine (EBM), decision-making about treatment of individual patients involves conscious, specific, and reasonable use of modern, best evidences (1). The purpose of EBM is ultimately to provide patients with the best treatment solutions. Thus, EBM helps avoid mistakes in the course of treatment and raises the quality and the cost-effectiveness of health care. Diagnosis and prognosis, two basic aspects of medicine and paramedicine, provide valuable information enabling patients and professionals to make decision. The results of diagnostic and prognostic processes must be as correct as possible, as they can have far-reaching consequences. The application of the EBM methods in diagnostic and prognostic processes used in healthcare is thus essential (2).

EBM requires from the physician the ability to search the medical literature and the skills in the interpretation of epidemiological and statistical results. However, evaluating the quality of a given study can be challenging in some cases, depending on the nature of the diagnostic test, the study design and statistics used. For instance, diagnostic tests involving measurable functional, biological or morphological changes of clinical significance usually obey to well established quality standards of reproducibility and validity and are relatively easy to compare based on their predictive values, sensitivity and specificity (3). In contrast, validity studies of tests in questionnaire format, commonly used for the diagnosis of mental and behavioral disorders, are more challenging to assess. Diagnostic questionnaire assessing mental disorders should obey to a number of methodological standards, such as psychometric properties, as part of its validation process (4). However, terms that denominate the psychometric properties have rather broad, sometimes vague definitions, while the statistical methods for their assessment vary widely across publications (4-11). Moreover, available methodological guidelines are heterogeneous and generally incomplete. Some of them are even contradictory (4, 6, 7). To date, no consensual methodological guideline exists for the whole validation process of mental health questionnaires and rating scales used for screening and diagnosis of mental disorders. The currently available standards focus on the methodological quality of single studies reporting diagnostic accuracy and psychometric properties. Examples of those standards are the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) (12) or the Standards Reporting of Diagnostic Accuracy Studies (STARD) (13). The Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) (14) is often used for the qualitative evidence appraisal in the systematic reviews. However, the latest is rather unhelpful from the statistical point of view.

This lack of harmonization regarding acceptable validity standards or criteria for various mental health questionnaires directly challenges the EBM application in diagnosis and subsequently in treatment of mental disorders, in particular, among non-specialized health professionals. In order to remedy this situation, we have established a general reference framework for the validation process of diagnostic tools for mental disorders, including self-reported measures of burnout. The burnout syndrome remains ill-defined and nosologically uncharacterized (15). Despite its increasing importance (16), burnout syndrome still has no consensual definition, which makes it difficult to manage. Maslach and Jackson (17) proposed the most prominent definition of burnout: a psychological syndrome that occurs in professionals who work with other people in challenging situations that is measured through three domains: 1-emotional exhaustion 2-depersonalisation and 3-personal accomplishment. From this definition, Maslach developed a first burnout measure: the Maslach Burnout Inventory (MBI). Apart from the MBI, a meta-analysis by O’Connor et al. (18) cited six other validated burnout measures: the Pines Burnout Measure (BM), the OLdenburg Burnout Inventory (OLBI), the Copenhagen Burnout Inventory (CBI), the Professional Quality of Life Scale (ProQOL III), the Psychologists Burnout Inventory (PBI), the Children’s Services Survey (CSS), and the Organizational Social Context Scale (OCS). Considering that psychological syndromes measures are heterogeneous, a closer look to the validation process of the currently used burnout measures should give insight on their legitimacy in medical practice and research.

Objectives

This article aims at presenting our methodological reference framework for the validation process of diagnostic tools for mental disorders as part of the protocol for our systematic review of burnout self-reported measures. The objectives of this systematic review are to assess the validation processes used in each of the selected burnout measures and to grade the evidence of the validity and psychometric quality of each burnout measure to select the most valid one(s) for use in medical practice and epidemiological research.

Reference framework for the validation process of diagnostic tools for mental disorders

This framework is provided in Supplementary material Table 1, organized in four columns, as follows: 1-psychometric validity criteria, 2-their definitions, 3-the methods commonly used to analyze them, and 4-the resulting statistical estimates and indices as well as the objective criteria for their respective interpretation. To construct this framework, we completed the demarche initiated by the French National Institute of Research on Security (INRS) for a comparative analysis of different scales and tools used for assessing psychosocial risks available in French language (4). First, we listed as exhaustively as possible the psychometric validity criteria and their definitions, using handbooks and published guidelines (4-11, 14, 15, 17-44). Second, we sorted the validity criteria, according to their most consensual denomination and definition and grouped them by sub-types according to Bolarinwa (6). Third, we filled the third and fourth columns of the table with appropriate analyses and indices’ interpretation for each validity criterion, using handbooks and published methodological guidelines (4-6, 8-11, 19-33, 35-44). Forth, we submitted the completed table of our framework to two independent experts with strong psychometric skills for critical review of the retained definitions, the completeness of the methods, and the appropriate choice of interpretation criteria. Finally, after discussion of the reviewers’ comments and getting consensus, we produced a current version of the framework. We consider it as a methodological referential because it allows non-specialized health professionals and researchers to understand and to correctly interpret the overall and specific validity criteria of a diagnostic tool for mental disorders, whatever the study design and statistical method used for its validation. Thanks to its multiple entries, it is possible to shift through validation studies by picking up terms about either validity criteria (20 criteria), analytical methods (21 methods) or the resulting indices and statistics grouped into 19 categories. Because of its analytical exhaustiveness and completeness for the three elements of the validation of diagnostic tests (i.e., validity, reproducibility and sensitivity), it constitutes a useful framework for quality appraisal of diagnostic tests for mental disorders.

Eligibility criteria

We will include 1-studies with quantitative methodology; 2-published in the original scientific article formats; 3-adressing the psychometric properties of at least one above-mentioned burnout measures in its original (not translated) version; 4-with sample size of at least of 100 participants. We will exclude 1-studies that do not meet the inclusion criteria; 2-studies for which no abstract and full text could be found; 3-studies where one of the eight burnout measures was used as a reference against another one, not included in this review; 4-studies where a translated version of burnout measure was used (e.g., translational validity and cross-cultural studies); 5-studies in which quantitative data on reliability or validity were missed; 6-studies where participants were not professionally employed (e.g., students, medical residents).

Search strategy

An experienced librarian will review the search strategy. It will consist of free-text words to specify three search strings: terms focusing on the burnout measure of interest (e.g., MBI), terms related to the validation of the measure, and a combination of the two first search strings results. Finally, one additional search string will consist of removing duplicates.

Study records

Data items

The extracted data will concern studies’ identification (i.e., authors, year of publication, journal, and title); samples’ characteristics (i.e., size, gender ratio, age, occupational activity, participation rate, representativity, burnout scores’ distribution); burnout measures’ characteristics (i.e., name, version, number of items, number of domains, domains’ names); and statistical methods used for assessing the psychometric properties outcome.

Outcomes and prioritization

The outcomes of interest will be the face validity, response validity, internal structure validity, convergent validity, discriminant validity, predictive validity, internal consistency, test-retest reliability, and alternate form reliability. Those criteria will enable to assess the psychometric properties used to validate the eight concerned burnout measures.

Risk of bias in individual studies

Two reviewers will independently assess the quality of each study using the COSMIN checklist (14). They will discuss any discrepancies, and they will resort to the arbitration of a third reviewer if needed.

Data synthesis