You are here

Article Text

Article menu

Download PDFPDF

XML
Original research
Measure what we want: a taxonomy of short generic person-reported outcome and experience measures (PROMs and PREMs)
  1. Tim Benson1,2
  1. 1R-Outcomes Ltd, Thatcham, UK
  2. 2Institute of Health Informatics, UCL, London, UK
  1. Correspondence to Dr Tim Benson; tim.benson{at}r-outcomes.com

Abstract

Introduction Health and care systems are complex and multifaceted, but most person-reported outcome and experience measures (PROMs and PREMs) address just one aspect. Multiple aspects need measuring to understand how what we do impacts patients, staff and services, and how these are affected by external factors. This needs survey tools that measure what people want, are valid, sensitive, quick and easy to use, and suitable for people with multiple conditions.

Methods We have developed a coherent family of short generic PROMs and PREMs that can be used in combination in a pick-and-mix way. Each measure has evolved iteratively over several years, based on literature review, user inputs and field testing. Each has has a common format with four items with four response options and is designed for digital data collection with standardised analytics and data visualisation tools. We focused on brevity and low reading age.

Results The results are presented in tabular format and as a taxonomy. The taxonomy is categorised by respondent type (patient or staff) and measure type. PROMs have subdomains: quality of life, individual care and community; PREMs have subdomains: service provided, provider culture and innovation. We show 22 patient-reported measures and 17 staff-reported measures. Previously published measures have been validated. Others are described for the first time.

Discussion and conclusions This family of measures is broad in scope but is not claimed to be comprehensive. Measures share a common look and feel, which enables common methods of data collection, reporting and data visualisation. They are used in service evaluation, quality improvement and as key performance indicators. The taxonomy helps to organise the whole, explain what each measure does and identify gaps and overlaps.

  • patient-reported outcome measures
  • patient satisfaction
  • surveys
  • attitude of health personnel
  • diffusion of innovation
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjoq-2019-000789

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Surveys, completed by patients or staff, are widely used in tailoring care, quality improvement, evaluation and population health management. They need to cover the things that matter most to those completing them and other stakeholders. The challenge is to do this in a simple easy-to-use way, while recognising the complexity inherent in the health domain.1

    Person-reported outcome measures (PROMs)2–4 and person-reported experience measures (PREMs) measure different things,5 with only weak correlation. PROMs measure people’s perception of their own situation; PREMs measure their perception of services provided. PROMs are a form of personal history and are of clinical value, but PREMs are usually anonymous, because people can be reluctant to criticise those they depend on. Individuals may choose to identify themselves in PREMs, but the default is not to.

    PROMs and PREMs may be condition-specific or generic. Two-thirds of health and care expenditure is for people living with three or more chronic conditions,6 but most PROMs apply to only one condition, which limits their use. Different measures have been developed independently and do not work well together.7 For example, in some measures a high score is good, in others high is bad. Scale ranges vary, such as 0–1, 0–10, 0–48 or 0–100.

    Generic measures work for all types of patients, treatments and conditions. They are based on the idea that people want similar things, such as good health and well-being, excellent service, supportive communities and organisations, care and innovations that meet their needs.

    Care quality is assessed in terms of structure, process and outcome.8 Our focus is on outcome as perceived by patients and staff. Perceived outcome is only one aspect of a complex whole, although broader than the traditional definitions of PROMs and PREMs.9 However, it does not cover all aspects of health outcomes, experience and patient-centred care.10

    Response rates are affected by perceived relevance and ease of use.11 Most measures require a higher reading age12 than the average reading age of the UK population, which is about 9 years.13

    Background

    This work has had a long gestation. During the 1970s, the author worked with Rachel Rosser to evaluate computer systems in a London hospital using a short staff-reported classification of disability and distress.14 Inter-rater reliability studies identified the importance of using clear, unambiguous wording.15

    During the mid-2000s, interest in PROMs and PREMs increased, as exemplified by Darzi’s NHS Next Stage Review High Quality Care for All, which recommended their wide use.16 Unfortunately, existing tools were not well suited to routine use, having been used mainly in pharmaceutical clinical trials, where respondents have few time limitations and only one condition.

    The author identified a need for a simple PROM that could be used on smartphones and tablets. This led to the development of the howRu health status measure, which evolved from Rosser’s classification. This was tested in a telephone survey of 2751 people living with long-term conditions, in comparison with 12-item Short Form Survey.17 It was also tested in comparison with 3-level version of EQ-5D in a hospital cardiovascular clinic,18 and in hip and knee replacement surgery.19

    After the Stafford Hospital scandal, the financial crash and change of government, political interest turned to patient experience (PREMs). The howRwe patient experience measure was developed along the same lines as howRu to be quick and easy-to-use routinely. It was tested in an orthopaedic presurgical assessment unit.20 The howRu and howRwe measures were both used in a census of 24 000 care home residents in the UK, Australia and New Zealand.21 22

    Person-centred care and new care models became a key focus during the mid-2010s. Wessex AHSN selected howRu and howRwe for use in the evaluation of the North East Hampshire and Farnham NHS Vanguard project, also known as Happy, Healthy at Home. This eventually used 17 different surveys with more than 2800 respondents. Explicit objectives included improved personal well-being and health confidence, which led to the development of the Personal Well-being Score (PWS) based on ONS4,23 and the Health Confidence Score (HCS).24Social prescribing and care navigation also attracted attention and evaluation funding, leading to related measures of loneliness, community cohesion and social determinants of health.

    During the same period, Wessex AHSN was tasked with evaluating and promoting the spread of digital health innovation, which stimulated the development of innovation adoption measures.25 These built on the author’s prior work about how spread26 27 and interoperability28 are impacted by both technical and non-technical factors (eg, culture).

    The aim of this paper is to describe the resulting family of generic measures, organised as a taxonomy. A taxonomy allows for measures to be viewed and compared, gaps identified and the body of work improved and developed further.

    Methods

    The author with colleagues has developed a family of short generic PROMs and PREMs to capture a broad range of patient and staff perceptions of quality of life, healthcare services, wider determinants of health, digital and service innovations. These measures share a common format and scoring scheme. They are picked and mixed as required to create longer surveys for different purposes in quality improvement, impact evaluation and as key performance indicators (KPIs).

    All measures are generic, suitable for most situations and clinical conditions, irrespective of case-mix, across health and social care. They can be completed on paper, smartphone, tablet, PC or via text message or voice.

    Using criteria set out in the literature,29–32 each measure was developed in a similar way.17 None of the work was commissioned formally or grant-funded. The author had full editorial control.

    In outline, the approach used was as follows:

    1. Recognise the need for a new measure, based on user feedback and other insights. All measures were developed to meet actual or perceived needs.

    2. Review the relevant literature and identify key themes.

    3. Develop prototypes, based on a common format of four items per measure and four response options per item.

    4. Discuss, revise and field test with users, colleagues and other stakeholders.

    5. Iterate, adapt, evolve and further test. This involved dozens or in some cases hundreds of iterations before all issues were resolved.

    6. Evaluate the measure for distribution (eg, skewness and kurtosis), internal reliability and construct validity.

    7. Publish in peer-review journal.

    The common format, with four items (questions) and four response options, is not a rigid rule and exceptions may be allowed to the number of items or options, although none is shown in this paper.

    The Health Confidence Score (figure 1) provides an example of the look and feel, showing the title, preamble and instructions, items (lines), options (columns), colour and emojis.24

    Figure 1
    Figure 1

    The Health Confidence Score: an example of a measure.

    Items

    Each item measures perception of one characteristic or theme in a measurement domain. Most domains have a well-understood ideal. Item wording needs to capture different aspects of the domain in ways that people readily understand.

    Particular attention was given to word count and readability. These were calculated using the word count and readability statistics included in Microsoft Word. In this paper, we use the text in the tables herein, including footnotes, with each item label treated as a separate sentence. The survey preamble and options are excluded, because the preamble is usually tailored to the local context and option repetition depends on administration mode (eg, the options should always be visible to the user). The readability measure is the Flesch Kincaid Grade (FKG), which estimates US school grade.33 As a guide, the reading age of a text is FKG plus five.

    Options

    The following option sets are used:

    • None, a little, quite a lot, extreme (none–extreme)

    • Strongly agree, agree, neutral, disagree (strongly agree–disagree)

    • Hardly ever, occasionally, sometimes, always (hardly ever–always)

    • Excellent, good, fair, poor (excellent–poor).

    This list is extensible. For example, we could also use:

    • Agree, neutral, disagree, strongly disagree (agree–strongly disagree)

    • Strongly agree, agree, disagree, strongly disagree (strongly agree–strongly disagree).

    Options are usually ordered left to right, from best to worst. We use colour coding and emoji (both of which are optional), from best (eg, green smiley face) to worst (eg, red sad face). Emoji are tailored to the meaning of each option set, using a choice from: grin, smile, neutral (straight mouth), unhappy and miserable.

    All items are optional. In most cases the recall period is now. Many PROMs use recall periods with questions such as: “how often have you experienced X” during the last week or month. However, many people find recall difficult (eg, most people find it hard to remember what they had for dinner 2 or 3 days ago).34 These measures avoid specifying a recall period other than today or yesterday.

    Scoring

    A high score is always good, which aids consistent understanding of results. This rule is followed even when the name of an item or measure implies that it measures something undesirable.

    For items about individuals, the scoring system is from 0 (worst) to 3 (best). For populations, the mean item score is transformed to a 0–100 scale using the formula: (mean item score)×100/3. For example, responding strongly agree to I know enough about my health scores 3 on the 0–3 individual scale and 100 on 0–100 population scale; disagree scores 0 on both scales.

    Most measures comprise a group of four items. A summary score is calculated for each measure as the sum of the item scores. Assuming four items, at the individual level this gives a 13-point scale from 0 (4×worst) to 12 (4×best). For populations, the mean summary score is shown on a scale from 0 to 100, using the formula: (mean summary score)×100/12. A summary score is not calculated if any item score is missing.

    Using a common 0–100 scale for item and summary mean scores enables direct comparison of the results. A mean score 100 occurs if all respondents chose the best option (the ceiling) and 0 if all chose the least desirable option (the floor). It is unlikely that an individual score will be confused with a population mean score, because they use different ranges.

    Taxonomy

    A taxonomy was developed as a way of organising and classifying the measures, to explain the range and scope of measures to others and to identify gaps and overlaps.

    A taxonomy is a system for classifying multifaceted, complex phenomena according to common conceptual domains and dimensions.35 It is a hierarchy of things or concepts in which each node (other than the root) has a single parent and any number of sibling and child nodes. Each node is a specialisation or sub-class of its parent (inheritance).

    The development of the taxonomy followed an iterative process similar to that used to develop its components. Key criteria were simplicity, coherence and inheritance.

    Patient and public involvement

    Many patients, health staff and members of the public took part in focus groups during the development of these measures. They helped test and refine early versions of the measures. Most focus groups were informal. Papers which describe the development and validation of specific measures provide more details of patient and public involvement for those measures.

    This paper does not report identifiable data about any individuals or groups.

    Results

    The taxonomy

    The results use the taxonomy as an organising principle or framework. Figure 2 shows the top levels.

    Figure 2
    Figure 2

    Top levels of the taxonomy. PREMs, person-reported experience measures; PROMs, person-reported outcome measures.

    Patient-reported and staff-reported measures cover the same domains, but there are important differences between them. It helps to consider these roles separately. Patients are subjects of care, but staff provide care (eg, clinicians, admin staff and volunteers) within an organisational structure. Staff see many patients and the data collection process is usually simpler. Many staff-reported measures were adapted from patient-reported measures.

    At the next level, the two broad categories of measure are person-reported outcome measures and person-reported experience measures.

    Person-reported outcome measures

    PROMs refer to the impact on individuals as perceived by the rater. They include measures of:

    • Quality of life

    • Individual care

    • Community

    Quality of life measures include people’s health status, personal wellbeing, fatigue and sleep patterns. These are usually about patients, recorded by patients themselves or proxies on their behalf.

    Individual care measures include health confidence, shared decision-making, self-care, behaviour change, adherence to treatment (eg, medication) and acceptance of loss. Individual care is typically based on interactions between patient and clinician (staff); both groups have their own perception of the outcome, which may differ.

    Community measures include external and environmental factors such as social determinants of health, loneliness, neighbour relationships and personal safety. This is mainly related to how and where people live.

    Person-reported experience measures

    PREMs measure people’s perception of the service provided. There are three domains:

    • Care provided

    • Provider culture

    • Innovation

    Care provided covers both individual services and the way that services work together. Patients and staff have views about the quality of care provided.

    Provider culture measures aspects of each health and care organisation’s policies and practice. Staff have more direct knowledge and experience of the culture than patients.

    Innovation focuses on the impact of specific innovations, such as digital health applications and new ways of working. Staff are invariably involved and patients less frequently.

    Tables

    Details of each measure are shown in tables 1–4.

    View this table:
    Table 1

    Patient-reported outcome measures

    View this table:
    Table 2

    Patient-reported experience measures

    View this table:
    Table 3

    Staff-reported outcome measures

    View this table:
    Table 4

    Staff-reported experience measures

    Each table is set out with six columns:

    1. Name: a short easy to understand name or label. The name is usually positively worded, but not always. For example, the health status (howRu) measure has an item for pain or discomfort. Here, the best (highest) score comes from having no pain. The English language is better at describing some aspects negatively.

    2. Options: the response options easure how much the respondent currently perceives some thing to be a problem. Many measures ask about agreement with positively worded statements using a scale from strongly agree to disagree.

    3. Text used in survey: text as presented to the respondent. In practice each survey also contains a preamble. This is not shown here, because it is usually context-specific and contains locally-specific instructions and context.

    4. Alias: a short unique alias name used in computer processing. This does not contain spaces; it uses UpperCamelCase to separate natural words and component parts.

    5. Words (FKG): the number of words and Flesch Kincaid readability grade.

    6. Notes: brief description and reference to a publication about each measure or the most influential source that influenced its development.

    Patient-reported measures

    Figure 3 shows patient-reported outcome and experience measures.

    Figure 3
    Figure 3

    Summary of patient-reported outcome (PROMs) and experience (PREMs) measures.

    Table 1 describes PROMs; table 2 describes PREMs.

    Staff-reported measures

    Staff-reported outcome and experience measures are summarised in figure 4.

    Figure 4
    Figure 4

    Summary of staff-reported outcome and experience measures. PROMs, patient-reported outcome measures; PREMs, patient-reported experience measures.

    Staff-reported outcome measures are described in table 3. Staff-reported experience measures are described in table 4.

    Table 5 summarises the number of measures by rater (patient-reported and staff-reported) and type (PROM or PREM). The expanded taxonomy is provided as a online supplementary file 1.

    Supplemental material

    : Expanded taxonomy of patient-reported and staff-reported R-Outcomes measures.
    View this table:
    Table 5

    Summary count of measures

    Discussion

    The need for generic measures with a broad scope is increasingly recognised, in particular for older people with long-term conditions.36 This taxonomy is, as far as we know, the most comprehensive, coherent framework or taxonomy of short generic measures that has been published. It is unusual in covering both patient-reported and staff-reported measures as well as PROMs and PREMs. It also covers external factors that affect health and well-being, and those that affect the spread of health innovations.

    A possible limitation of our approach is that it is based primarily on the work of a single author. The measures were not developed as part of a grant-funded research programme in an academic setting, nor for use in clinical trials. Some people may consider this to be a strength on the basis that theories should emerge from bottom-up, empirical experimentation. However, each measure has been strongly influenced by existing theories and paradigms.

    Four response options may also be regarded as a limitation, but this is not our experience. The best option (the ceiling) can be thought of as being as good as it gets. If used appropriately this does not produce a ceiling effect, whereby the measure is unable to detect valuable improvements. A floor effect (the worst option) is more problematic, because things can always get worse. In general, if a respondent is at the floor, this calls for remedial action. Intermediate options can be regarded as being less good than the ceiling and less bad than the floor, respectively.

    Answering any survey question involves four cognitive steps: (1) understand the question; (2) retrieve relevant information from memory; (3) judge which response option fits best and (4) responding in a way that fits the judgement. There is always a risk that raters may satisfice by doing one or more of these suboptimally, to save effort. This can give rise to a number of effects such as acquiescence bias, primacy effect and non-differentiation. This risk is greater in surveys answered in private, where there is no other person present to sense-check the responses, if a survey is long or difficult, seen as a chore or not regarded as relevant.37

    The response options form an ordinal scale, which suggests that non-parametric statistics should be used.38 However, interval or ratio scales ares needed for health economic calculations, such as quality-adjusted life year or Load calculations.39 We have explored the generation of multi-attribute interval weightings using pairwise comparisons with the PAPRIKA(Potentially All Pairwise RanKings of all possible Alternatives) method.40 In the absence of such weightings, we ascribe unweighted integer values to these options to calculate mean scores for item and summary scores of populations. In ideal situations (eg, people in good health), the distributions of these measures are skewed to the top, but summary scores for people with long-term conditions show a distribution which is close to normal.17 23 24 In practice, we find that parametric and non-parametric statistical tests produce very similar results.

    It is useful to identify the minimally important difference (MID) between two sets of measurements. Half a SD is a widely used criterion at the individual level.41 So, for a summary score, if SD=20 on 0–100 scale (which is typical), the MID=0.5 (SD)=10. For populations, sample size (n) is a key variable, so if n=64 and SD=20 the 95% CI is ±1.96(SD/√n)=±4.9.

    Carers or informal care givers form a special case, sharing aspects of both patients and staff; they are not discussed here, but will be considered in a future paper. There is no prohibition on people using measures that are not explicitly designed for them. For example, any measure may be completed by a proxy, but if so this should be recorded.

    Four of these measures (health status, health confidence, personal well-being and experience) have been validated psychometrically at the time of writing.17 20 24 24 Five have been described in the literature (digital confidence, user satisfaction, innovation readiness, innovation process and behaviour change),25 three have been described in the specific context of residential care homes (work well-being, job confidence and service provided)42 and the process is underway for others. We encourage other validation studies.

    Practical implementation always needs to consider the whole end-to-end process, not only what measures to use and why, but who, where, when and how.43 This includes ensuring that people are asked to complete surveys, that all the stakeholders involved understand what is being asked and why and that all aspects of survey management, including supporting technology and analytics, are properly resourced. Results may be reported at the individual level to tailor individual care, or aggregated to measure the performance of specific services or user needs.

    These measures have been used with success in commissioning services and in the evaluation of new care models,44 social prescribing,23 care home services21 22 42 and in digital health evaluation, including self-care for people with diabetes and detection of atrial fibrillation (AF).

    Innovation measures have been mapped to the Nonadoption, Abandonment and failure to Scale-up, Spread and Sustain framework (NASSS), which uses the lens of complexity theory to explain and avoid failures of digital health innovations.45

    PROMs may be thought of as patient history, form part of the clinical record and inform patient care. However, identifiable data are subject to strict information governance, requiring compliance with the General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPAA) and similar laws and regulations.46 In practice, to avoid issues of information governance, many PROMs are collected anonymously.

    Widespread use of PROMs and PREMs requires integration with electronic health records and other health IT systems. This needs semantic interoperability using standards such as Fast Health Interoperability Resources (FHIR) and coding schemes such as Logical Observation Identifiers and Codes (LOINC) and Systematised Nomenclature of Medicine Clinical Terms (SNOMED CT).28 47 FHIR Questionnaire and Questionnaire Response resources support the use of surveys in day-to-day care and clinical research.48 LOINC supports the structure and content of assessment surveys.49 LOINC and SNOMED CT (UK Edition) codes have been allocated for some measures (eg, howRu and HCS)17 24 and applications for the others are underway.

    Conclusions

    This paper describes a family of generic PROMs and PREMs for routine use and in evaluation. This family of measures has a broad scope but is not claimed to be comprehensive.

    The measures are described in tables and organised as a taxonomy. The taxonomy is categorised by respondent (patient or staff) and type (PROMs or PREMs). We describe 22 patient measures and 17 staff measures. Some are described here for the first time. These measures may be used to help tailor individual care, and at aggregate level for evaluation and accountability.

    PROMs are grouped under categories for quality of life, individual care and community. PREMs have categories for service provided, provider culture and innovation. All of the measures share the same form, with four items with four response options. The measures are short with low reading age. They can be used to build short questionnaires for different purposes, using common survey management, data analyics, data visualisation and reporting tools. This flexibility allows practitioners to select measures on a pick-and-mix basis to meet their local needs.

    Lay summary

    This paper describes a family of short generic PROMs and PREMs, designed for use in combinations in a pick-and-mix way. PROMs cover quality of life, individual care and community; PREMs cover service provided, provider culture and innovation. Common properties of these measures include specialty-independence, brevity, ease of use, low reading age, a common format, data collection, reporting and data visualisation methods. They are used in tailoring care, quality improvement, service evaluation and as KPIs.

    Acknowledgments

    The author would like to thank all the people who have contributed to the development of these measures and this taxonomy. In particular, to members of the Insight team at Wessex AHSN, Dr Helen Seers and the anonymous reviewers who have helped improve this paper greatly.

    References

    1. ↵
      1. Plsek PE,
      2. Greenhalgh T
      . The challenge of complexity in health care. Br Med J2001;323:625–8.
      OpenUrlFREE Full Text
    2. ↵
      1. Kingsley C,
      2. Patel S
      . Patient-reported outcome measures and patient-reported experience measures. BJA Education2017;17:137–44.doi:10.1093/bjaed/mkw060
      OpenUrl
    3. ↵
      1. Black N
      . Patient reported outcome measures could help transform healthcare. BMJ2013;346:f167.doi:10.1136/bmj.f167
    4. ↵
      1. Calvert M,
      2. Kyte D,
      3. Price G, et al
      . Maximising the impact of patient reported outcome assessment for patients and society. BMJ2019;364:k5267.doi:10.1136/bmj.k5267
      OpenUrlFREE Full Text
    5. ↵
      1. Black N,
      2. Varaganum M,
      3. Hutchings A
      . Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery. BMJ Qual Saf2014;23:534–42.doi:10.1136/bmjqs-2013-002707
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Buttorff C,
      2. Ruder T,
      3. Bauman M
      . Multiple chronic conditions in the United States. Santa Monica: RAND Health, 2017.
    7. ↵
      1. Macefield RC,
      2. Jacobs M,
      3. Korfage IJ, et al
      . Developing core outcomes sets: methods for identifying and including patient-reported outcomes (PROs). Trials2014;15:49.doi:10.1186/1745-6215-15-49
    8. ↵
      1. Donabedian A
      . The quality of care. How can it be assessed?JAMA1988;260:1743–8.doi:10.1001/jama.260.12.1743
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Devlin NJ,
      2. Appleby J
      . Getting the most out of PROMs. Putting health outcomes at the heart of NHS decision-making. London: The King's Fund, 2010.
    10. ↵
      1. de Silva D
      . Helping measure person-centred care. London: Health Foundation, 2014.
    11. ↵
      1. Jahagirdar D,
      2. Kroll T,
      3. Ritchie K, et al
      . Using patient reported outcome measures in health services: a qualitative study on including people with low literacy skills and learning disabilities. BMC Health Serv Res2012;12:431.doi:10.1186/1472-6963-12-431
    12. ↵
      1. Paz SH,
      2. Liu H,
      3. Fongwa MN, et al
      . Readability estimates for commonly used health-related quality of life surveys. Qual Life Res2009;18:889–900.doi:10.1007/s11136-009-9506-y
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Richards S
      . Specifying a reading age for web content. Content design London, 2016. Available: https://contentdesign.london/reading/specifying-a-reading-age-for-web-content [Accessed 3 Sep 2019].
    14. ↵
      1. Rosser R,
      2. Benson T
      . New tools for evaluation: their application to computers. In: Anderson J, ed. Medical informatics Europe 78, first Congress of the European Federation for medical informatics: proceedings. Cambridge, England: Springer Verlag, 1978: 701–10.
    15. ↵
      1. Benson TJ
      . Classification of disability and distress by ward nurses: a reliability study. Int J Epidemiol1978;7:359–61.doi:10.1093/ije/7.4.359
      OpenUrlCrossRefPubMed
    16. ↵
      1. Darzi A
      . High quality care for all: NHS next stage review final report. London: Stationery Office, 2008.
    17. ↵
      1. Benson T,
      2. Sizmur S,
      3. Whatling J,
      4. Arikan S, et al
      . Evaluation of a new short generic measure of HRQoL: howRu. Inform Prim Care2010;18:89–101.doi:10.14236/jhi.v18i2.758
      OpenUrlPubMed
    18. ↵
      1. Benson T,
      2. Potts HWW,
      3. Whatling JM, et al
      . Comparison of howRU and EQ-5D measures of health-related quality of life in an outpatient clinic. Inform Prim Care2013;21:12–17.doi:10.14236/jhi.v21i1.9
      OpenUrlPubMed
    19. ↵
      1. Benson T,
      2. Williams DH,
      3. Potts HWW
      . Performance of EQ-5D, howRu and Oxford hip & knee scores in assessing the outcome of hip and knee replacements. BMC Health Serv Res2016;16:512.doi:10.1186/s12913-016-1759-x
    20. ↵
      1. Benson T,
      2. Potts HWW
      . A short generic patient experience questionnaire: howRwe development and validation. BMC Health Serv Res2014;14:499.doi:10.1186/s12913-014-0499-z
    21. ↵
      1. Benson T,
      2. Bowman C
      . Health status of care home residents: practicality and construct validity of data collection by staff at scale. BMJ Open Qual2019;8:e000704.doi:10.1136/bmjoq-2019-000704
    22. ↵
      1. Benson T,
      2. Bowman C
      . Comparison of staff and resident health status ratings in care homes. BMJ Open Qual2020;9:e000801.
    23. ↵
      1. Benson T,
      2. Sladen J,
      3. Liles A, et al
      . Personal Wellbeing Score (PWS)-a short version of ONS4: development and validation in social prescribing. BMJ Open Qual2019;8:e000394.doi:10.1136/bmjoq-2018-000394
    24. ↵
      1. Benson T,
      2. Potts HWW,
      3. Bark P, et al
      . Development and initial testing of a Health Confidence Score (HCS). BMJ Open Qual2019;8:e000411.doi:10.1136/bmjoq-2018-000411
    25. ↵
      1. Benson T
      . Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change. BMJ Health Care Inform2019;26:e000018doi:10.1136/bmjhci-2019-000018
    26. ↵
      1. Benson T
      . Why general practitioners use computers and hospital doctors do not--Part 1: incentives. BMJ2002;325:1086–9.doi:10.1136/bmj.325.7372.1086
      OpenUrlFREE Full Text
    27. ↵
      1. Benson T
      . Why general practitioners use computers and hospital doctors do not--Part 2: scalability. BMJ2002;325:1090–3.doi:10.1136/bmj.325.7372.1090
      OpenUrlFREE Full Text
    28. ↵
      1. Benson T,
      2. Grieve G
      . Principles of health Interoperability: SNOMED CT, HL7 and FHIR. 3rd edn. London: Springer, 2016.
    29. ↵
      1. Streiner DL,
      2. Norman GR,
      3. Cairney J
      . Health measurement scales: a practical guide to their development and use. 5th edn. Oxford University Press, 2015.
    30. ↵
      1. Dillman DA,
      2. Smyth JD,
      3. Christian LM
      . Internet, phone, mail, and mixed-mode surveys: the tailored design method. 4th edn. Hoboken NJ: John Wiley & Sons, 2014.
    31. ↵
      1. Boateng GO,
      2. Neilands TB,
      3. Frongillo EA, et al
      . Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health2018;6:149.doi:10.3389/fpubh.2018.00149
      OpenUrl
    32. ↵
      1. Mokkink LB,
      2. Terwee CB,
      3. Patrick DL, et al
      . The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res2010;19:539–49.doi:10.1007/s11136-010-9606-8
      OpenUrlCrossRefPubMedWeb of Science
    33. ↵
      1. Kincaid JP,
      2. Fishburne Jr RP, et al
      . Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for Navy enlisted personnel. naval technical training command Millington Tn research branch1975.
    34. ↵
      1. Stull DE,
      2. Leidy NK,
      3. Parasuraman B, et al
      . Optimal recall periods for patient-reported outcomes: challenges and potential solutions. Curr Med Res Opin2009;25:929–42.doi:10.1185/03007990902774765
      OpenUrlCrossRefPubMed
    35. ↵
      1. Bradley EH,
      2. Curry LA,
      3. Devers KJ
      . Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res2007;42:1758–72.doi:10.1111/j.1475-6773.2006.00684.x
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Murphy M,
      2. Hollinghurst S,
      3. Cowlishaw S, et al
      . Primary care outcomes questionnaire: psychometric testing of a new instrument. Br J Gen Pract2018;68:e433–40.doi:10.3399/bjgp18X695765
      OpenUrlAbstract/FREE Full Text
    37. ↵
      1. Krosnick JA
      . Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol1991;5:213–36.doi:10.1002/acp.2350050305
      OpenUrlCrossRefWeb of Science
    38. ↵
      1. Siegel S
      . Nonparametric statistics for the behavioural sciences. New York: McGraw-Hill, 1956.
    39. ↵
      1. Benson T
      . The load model: an alternative to QALY. J Med Econ2017;20:107–13.doi:10.1080/13696998.2016.1229198
      OpenUrl
    40. ↵
      1. Hansen P,
      2. Ombler F
      . A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. Journal of Multi-Criteria Decision Analysis2008;15:87–107.doi:10.1002/mcda.428
      OpenUrl
    41. ↵
      1. Norman GR,
      2. Sloan JA,
      3. Wyrwich KW
      . Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care2003;41:582–92.doi:10.1097/01.MLR.0000062554.74615.4C
      OpenUrlCrossRefPubMedWeb of Science
    42. ↵
      1. Benson T,
      2. Sladen J,
      3. Done J, et al
      . Monitoring work well-being, job confidence and care provided by care home staff using a self-report survey. BMJ Open Qual2019;8:e000621.doi:10.1136/bmjoq-2018-000621
    43. ↵
      1. Basch E
      . Patient-Reported Outcomes - Harnessing Patients' Voices to Improve Clinical Care. N Engl J Med2017;376:105–8.doi:10.1056/NEJMp1611252
      OpenUrlCrossRefPubMed
    44. ↵
      1. Liles A,
      2. Darnton P,
      3. Sibley A, et al
      . How we are evaluating the impact of new care models on how people feel in Wessex, 2017. Available: http://wessexahsn.org.uk/img/news/Evaluating Patient Outcomes in Wessex.pdf
    45. ↵
      1. Greenhalgh T,
      2. Wherton J,
      3. Papoutsi C, et al
      . Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res2017;19:e367.doi:10.2196/jmir.8775
    46. ↵
      1. Phillips M
      . International data-sharing norms: from the OECD to the general data protection regulation (GDPR). Hum Genet2018;137:575–82.doi:10.1007/s00439-018-1919-7
      OpenUrl
    47. ↵
      1. Mandl KD,
      2. Gottlieb D,
      3. Ellis A
      . Beyond one-off integrations: a commercial, substitutable, reusable, Standards-Based, electronic health Record-Connected APP. J Med Internet Res2019;21:e12902.doi:10.2196/12902
    48. ↵
      1. Leroux H,
      2. Metke-Jimenez A,
      3. Lawley MJ
      . Towards achieving semantic interoperability of clinical study data with FHIR. J Biomed Semantics2017;8:41.doi:10.1186/s13326-017-0148-7
    49. ↵
      1. Bakken S,
      2. Cimino JJ,
      3. Haskell R, et al
      . Evaluation of the clinical LOINC semantic structure as a terminology model for standardized assessment measures. J Am Med Inform Assoc2000;7:529–38.
      OpenUrlCrossRefPubMed
      1. Walker M
      . Why we sleep. London: Allen Lane, 2017.
      1. Sharpe M,
      2. Wilks D
      . Fatigue. Br Med J2002;325:480–3.
      OpenUrlFREE Full Text
      1. Shrivastava SR,
      2. Shrivastava PS,
      3. Ramasamy J
      . Role of self-care in management of diabetes mellitus. J Diabetes Metab Disord2013;12:14.doi:10.1186/2251-6581-12-14
      OpenUrlPubMed
      1. Barry MJ,
      2. Edgman-Levitan S
      . Shared decision making--pinnacle of patient-centered care. N Engl J Med2012;366:780–1.doi:10.1056/NEJMp1109283
      OpenUrlCrossRefPubMedWeb of Science
      1. Michie S,
      2. van Stralen MM,
      3. West R
      . The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci2011;6:42.doi:10.1186/1748-5908-6-42
      OpenUrlCrossRefPubMed
      1. Lam WY,
      2. Fresco P
      . Medication adherence measures: an overview. Biomed Res Int2015;2015:1–12.doi:10.1155/2015/217047
      OpenUrlCrossRef
      1. Kübler-Ross E
      . On death and dying. New York: Simon & Schuster, 1969.
      1. Marmot M
      . The health gap: the challenge of an unequal world. Lancet2015;386:2442–4.doi:10.1016/S0140-6736(15)00150-6
      OpenUrlCrossRefPubMed
      1. Goodman A
      . Measuring your impact on loneliness in later life. London: Campaign to end loneliness, 2015.
      1. Putnam RD
      . Bowling alone: the collapse and revival of American community. New York: Simon and Schuster, 2001.
      1. Bilsky W
      . Fear of crime, personal safety and well-being: a common frame of reference. Universitäts-und Landesbibliothek Münster, 2017.
      1. Snape D,
      2. Martin G
      . Measuring loneliness – guidance for use of the national indicators on surveys. Office for National Statistics, 2018.
      1. NHS
      . The NHS long term plan. London: NHS, 2019. www.longtermplan.nhs.uk
      1. van Staa T-P,
      2. Goldacre B,
      3. Buchan I, et al
      . Big health data: the need to earn public trust. Br Med J2016;354:i3636.doi:10.1136/bmj.i3636
      OpenUrlFREE Full Text
      1. Prensky M
      . Digital natives, digital immigrants Part 1. On the Horizon2001;9:1–6.doi:10.1108/10748120110424816
      OpenUrlCrossRef
      1. Pappas N
      . Marketing strategies, perceived risks, and consumer trust in online buying behaviour. Journal of Retailing and Consumer Services2016;29:92–103.doi:10.1016/j.jretconser.2015.11.007
      OpenUrl
      1. Stoyanov SR,
      2. Hides L,
      3. Kavanagh DJ, et al
      . Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth2015;3:e27.doi:10.2196/mhealth.3422
      1. Rogers EM
      . Diffusion of innovations. 5th edn. The Free Press, 2003.
      1. Algase DL,
      2. Beck C,
      3. Kolanowski A, et al
      . Need-driven dementia-compromised behavior: an alternative view of disruptive behavior. Am J Alzheimers Dis1996;11:10–19.doi:10.1177/153331759601100603
      OpenUrl
      1. Gittell JH
      . Transforming relationships for high performance: the power of relational coordination. Stanford Business Books, 2016.
      1. Gandhi TK,
      2. Kaplan GS,
      3. Leape L, et al
      . Transforming concepts in patient safety: a progress report. BMJ Qual Saf2018;27:1019–26.doi:10.1136/bmjqs-2017-007756
      OpenUrlAbstract/FREE Full Text
      1. Privitera M,
      2. Weisman R,
      3. Cerulli C, et al
      . Violence toward mental health staff and safety in the work environment. Occup Med2005;55:480–6.doi:10.1093/occmed/kqi110
      OpenUrlCrossRefPubMed
      1. May CR,
      2. Mair F,
      3. Finch T, et al
      . Development of a theory of implementation and integration: normalization process theory. Implement Sci2009;4:29.doi:10.1186/1748-5908-4-29
      OpenUrlCrossRefPubMed

    Footnotes

    • Twitter @timbenson

    • Contributors The author takes full responsibility for this manuscript.

    • Funding There was no specicific grant for this research from any funding agency in the public, commercial or non-profit sectors.

    • Competing interests TB is a director and shareholder in R-Outcomes Ltd, which provides survey and evaluation services using these measures. Please contact R-Outcomes Ltd if you wish to use these measures (https://r-outcomes.com).

    • Patient and public involvement Patients and/or the public were involved in the design, conduct, reporting or dissemination plans of this research. Refer to the 'Methods' section for further details.

    • Patient consent for publication Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data availability statement Data sharing not applicable as no datasets generated and/or analysed for this study.