Abstract
Objective
Physiological track-and-trigger warning systems are used to identify patients on acute wards at risk of deterioration, as early as possible. The objective of this study was to assess the inter-rater and intra-rater reliability of the physiological measurements, aggregate scores and triggering events of three such systems.
Design
Prospective cohort study.
Setting
General medical and surgical wards in one non-university acute hospital.
Patients and participants
Unselected ward patients: 114 patients in the inter-rater study and 45 patients in the intra-rater study were examined by four raters.
Measurements and results
Physiological observations obtained at the bedside were evaluated using three systems: the medical emergency team call-out criteria (MET); the modified early warning score (MEWS); and the assessment score of sick-patient identification and step-up in treatment (ASSIST). Inter-rater and intra-rater reliability were assessed by intra-class correlation coefficients, kappa statistics and percentage agreement. There was fair to moderate agreement on most physiological parameters, and fair agreement on the scores, but better levels of agreement on triggers. Reliability was partially a function of simplicity: MET achieved a higher percentage of agreement than ASSIST, and ASSIST higher than MEWS. Intra-rater reliability was better then inter-rater reliability. Using corrected calculations improved the level of inter-rater agreement but not intra-rater agreement.
Conclusion
There was significant variation in the reproducibility of different track-and-trigger warning systems. The systems examined showed better levels of agreement on triggers than on aggregate scores. Simpler systems had better reliability. Inter-rater agreement might improve by using electronic calculations of scores.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Department of Health and NHS Modernisation Agency (2003) The National Outreach Report. Department of Health, London
Lee A, Bishop G, Hilman K, Daffurn K (1995) The medical emergency team. Anaesth Intensive Care 23:183–186
Subbe CP, Kruger M, Rutherford P, Gemmell L (2001) Patients at risk: validation of a modified Early Warning Score in medical admissions. Q J Med 94:521–526
Buist MD, Moore GE, Bernard SA, Waxman BP, Anderson JN, Nguyen TV (2002) Effects of a medical emergency team on reduction of incidence of and mortality from unexpected cardiac arrests in hospital: preliminary study. Br Med J 324:387–390
Pittard AJ (2003) Out of our reach? Assessing the impact of introducing a critical care outreach service. Anaesthesia 58:882–885
Stenhouse C, Coates S, Tivey M, Allsop P, Parker T (2000) Prospective evaluation of a Modified Early Warning Score to aid earlier detection of patients developing critical illness on a general surgical ward. Br J Anaesth 84:663P
Subbe CP, Hibbs R, Williams E, Rutherford P, Gemmel L (2002) ASSIST: a screening tool for critically ill patients on general medical wards. Intensive Care Med 28:S21
Harrison D (2004) KAPUTIL: Stata module to generate confidence intervals and sample size calculations for the kappa-statistic. Statistical Software Components S446501, Boston College Department of Economics. (Available at http://ideas.repec.org/c/boc/bocode/s446501.html)
Armitage P, Berry G, Matthews JNS (2002) Statistical methods in medical research. Blackwell, Oxford, pp 704–707
Meade MO, Cook RJ, Guyatt GH, Groll R, Kachura JR, Bedard M, Cook DJ, Slutsky AS, Stewart TE (2000) Interobserver variation in interpreting chest radiographs for the diagnosis of acute respiratory distress syndrome. Am J Respir Crit Care Med 161:85–90
Altman D (1991) Practical statistics for medical research. Chapman and Hall, London, pp 401–409
Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell L (2003) Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia 58:797–802
Giuliano KK, Scott SS, Elliot S, Giuliano AJ (1999) Temperature measurement in critically ill orally intubated adults: a comparison of pulmonary artery core, tympanic, and oral methods. Crit Care Med 27:2188–2193
Polderman KH, Christiaans HMT, Wester JP, Spijkstra JJ, Girbes ARJ (2001) Intra-observer variability in APACHE II scoring. Intensive Care Med 27:1550–1552
Polderman KH, Jorna EMF, Girbes ARJ (2001) Inter-observer variability in APACHE II scoring: effect of strict guidelines and training. Intensive Care Med 27:1365–1369
Chen LM, Martin CM, Morrison TL, Sibbald WJ (1999) Interobserver variability in data collection of the APACHE II score in teaching and community hospitals. Crit Care Med 27:1999–2004
Rué M, Valero C, Quintana S, Artigas A, Álvarez M (2000) Interobserver variability of the measurement of the mortality probability models (MPM II) in the assessment of severity of illness. Intensive Care Med 26:286–291
Lefering R, Zart M, Neugebauer EAM (2000) Retrospective evaluation of the simplified therapeutic intervention scoring system (TISS-28) in a surgical intensive care unit. Intensive Care Med 26:1794–1802
Arts DGT, de Keizer NF, Vroom MB, de Jonge E (2005) Reliability and accuracy of Sequential Organ Failure Assessment (SOFA) scoring. Crit Care Med 33:1988–1993
Acknowledgements
This study was funded by the UK National Health Service Research and Development Service Delivery and Organisation Programme (SDO/74/2004). The authors thank S. Ameeth, S. Collins, K. Ghosh, C. Rincon and J. Tobler for their help in preparing the study, obtaining consent from patients and collecting the data. We thank A. Pawley for entering data into electronic format and L. Gemmell for advising on the format and facilitating the setup of the study.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Subbe, C.P., Gao, H. & Harrison, D.A. Reproducibility of physiological track-and-trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 33, 619–624 (2007). https://doi.org/10.1007/s00134-006-0516-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00134-006-0516-8