Abstract
In economic evaluation, mathematical models have a central role as a way of integrating all the relevant information about a disease and health interventions, in order to estimate costs and consequences over an extended time horizon. Models are based on scientific knowledge of disease (which is likely to change over time), simplifying assumptions and input parameters with different levels of uncertainty; therefore, it is sensible to explore the consistency of model predictions with observational data. Calibration is a useful tool for estimating uncertain parameters, as well as more accurately defining model uncertainty (particularly with respect to the representation of correlations between parameters). Calibration involves the comparison of model outputs (e.g. disease prevalence rates) with empirical data, leading to the identification of model parameter values that achieve a good fit.
This article provides guidance on the theoretical underpinnings of different calibration methods. The calibration process is divided into seven steps and different potential methods at each step are discussed, focusing on the particular features of disease models in economic evaluation. The seven steps are (i) Which parameters should be varied in the calibration process? (ii) Which calibration targets should be used? (iii) What measure of goodness of fit should be used? (iv) What parameter search strategy should be used? (v) What determines acceptable goodness-of-fit parameter sets (convergence criteria)? (vi) What determines the termination of the calibration process (stopping rule)? (vii) How should the model calibration results and economic parameters be integrated?
The lack of standards in calibrating disease models in economic evaluation can undermine the credibility of calibration methods. In order to avoid the scepticism regarding calibration, we ought to unify the way we approach the problems and report the methods used, and continue to investigate different methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Drummond MFSM, Torrance GW, et al. Methods for the economic evaluation of health care programmes. 3rd ed. New York: Oxford University Press, 2005
Box G, Draper N. Empirical model-building and response surfaces. 1st ed. New York: John Wiley & Sons, 1987
Philips Z, Ginnelly L, Sculpher M, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess 2004 Sep; 8 (36): iii-iiv, ix-xi, 1–158
Cooper BS. Confronting models with data. J Hosp Infect 2007 Jun; 65 Suppl. 2: 88–92
Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics 2006; 24 (11): 1043–53
Van de Velde N, Brisson M, Boily M-C. Modeling human papillomavirus vaccine effectiveness: quantifying the impact of parameter uncertainty. Am J Epidemiol 2007 Apr 1; 165 (7): 762–75
Jit M, Choi YH, Edmunds WJ. Economic evaluation of human papillomavirus vaccination in the United Kingdom. BMJ 2008 Jul 17; 337: a769
Stout NK, Knudsen AB, Kong CY, et al. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics 2009; 27 (7): 533–45
National Institute for Health and Clinical Excellence. Guide to the methods of technology appraisal. London: NICE, 2008 [online]. Available from URL: http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf [Accessed 2009 Jun 23]
Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 3rd ed. Ottawa (ON): CADTH, 2006 [online]. Available from URL: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf [Accessed 2009 Jun 23]
Pharmaceutical Benefits Advisory Committee. 1995 guidelines for the pharmaceutical industry on preparation of submission to the Pharmaceutical Benefits Advisory Committee. Woden (SA): PBAC, 1995 [online]. Available from URL: http://www.health.gov.au/internet/main/publishing.nsf/Content/health-pbs-general-pubs-pharmpac-part1.htm [Accessed 2009 Jun 23]
Weinstein MC, O’Brien B, Hornberger J, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices–Modeling Studies. Value Health 2003 Jan-Feb; 6 (1): 9–17
Drummond MF, McGuire A. Economic evaluation in health care: merging theory with practice. 1st ed. New York: Oxford University Press, 2001
Karnon J, Vanni T. Calibrating models in economic evaluation: a comparison of alternative measures of goodnessof-fit, parameter search strategies and convergence criteria. Pharmacoeconomics 2011; 29 (1): 51–62
Ades AE, Claxton K, Sculpher M. Evidence synthesis, parameter correlation and probabilistic sensitivity analysis. Health Econ 2006; 15: 373–81
Kim JJ, Kuntz KM, Stout NK, et al. Multiparameter calibration of a natural history model of cervical cancer. Am J Epidemiol 2007 Jul 15; 166 (2): 137–50
Kim L, Thompson S. Uncertainty and validation of health economic decision models. Health Econ 2010; 19 (1): 43–55
Popper K. Conjectures and refutations: the growth of scientific knowledge. London: Routledge, 1963
Foss AM, Watts CH, Vickerman P, et al. Could the CARESHAKTI intervention for injecting drug users be maintaining the low HIV prevalence in Dhaka, Bangladesh? Addiction 2007; 102 (1): 114–25
Williams JR, Foss AM, Vickerman P, et al. What is the achievable effectiveness of the India AIDS initiative intervention among female sex workers under target coverage? Model projections from southern India. Sex Transm Infect 2006; 82 (5): 372–80
Goldhaber-Fiebert JD, Stout NK, Ortendahl J, et al. Modeling human papillomavirus and cervical cancer in the United States for analyses of screening and vaccination. Popul Health Metr 2007; 5 (1): 11
Jit M, Gay N, Soldan K, et al. Estimating progression rates for human papillomavirus infection from epidemiological data. Med Decis Making 2010 Jan-Feb; 306 (1): 84–98
Suárez E, Smith JS, Bosch FX, et al. Cost-effectiveness of vaccination against cervical cancer: a multi-regional analysis assessing the impact of vaccine characteristics and alternative vaccination scenarios. Vaccine 2008; 26 Suppl. 5: F29–45
Berkhof J, Bruijne MCD, Zielinski GD, et al. Evaluation of cervical screening strategies with adjunct high-risk human papillomavirus testing for women with borderline or mild dyskaryosis. Int J Cancer 2006; 118 (7): 1759–68
Welton NJ, Ades AE. Estimation of markov chain transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis, and model calibration. Med Decis Making 2005 Nov-Dec; 25 (6): 633–45
Anderson R, Haas M, Shanahan M. The cost-effectiveness of cervical screening in Australia: what is the impact of screening at different intervals or over a different age range? Aust N Z J Public Health 2008; 32 (1): 43–52
Mandelblatt J, Schechter CB, Lawrence W, et al. The SPECTRUM population model of the impact of screening and treatment on US breast cancer trends from 1975 to 2000: principles and practice of the model methods. J Natl Cancer Inst Monogr 2006; (36): 47–55
Karnon J, Goyder E, Tappenden P, et al. A review and critique of modelling in prioritising and designing screening programmes. Health Technol Assess 2007 Dec; 11 (52): iii-iv, ix-xi, 1–145
Pickles M, Foss AM, Vickerman P, et al. Interim modelling analysis to validate reported increases in condom use and assess HIV infections averted among female sex workers and clients in southern India following a targeted HIV prevention programme. Sex Transm Infect 2010 Feb; 86 Suppl. 1: i33–43
McMahon PM, Kong CY, Weinstein MC, et al. Adopting helical CT screening for lung cancer. Cancer 2008; 113 (12): 3440–9
Kim JJ, Wright TC, Goldie SJ. Cost-effectiveness of human papillomavirus DNA testing in the United Kingdom, the Netherlands, France, and Italy. J Natl Cancer Inst 2005; 97 (12): 888–95
Press WH, Teukolsky SA, Vetterling WT, et al. Numerical recipes in C: the art of scientific computing. 2nd ed. Cambridge (NY): Cambridge University Press, 1992
Trotter CL, Edmunds WJ. Modelling cost effectiveness of meningococcal serogroup C conjugate vaccination campaign in England and Wales. BMJ 2002; 324 (7341): 809
Keeling M. Modeling infectious diseases in humans and animals. 1st ed. Princeton (NJ): Princeton University Press, 2007
Rao SS. Engineering optimization: theory and practice. 4th ed. Chichester: John Wiley & Sons, 2009
Freitas AA. A critical review of multi-objective optimization in data mining: a position paper. SIGKDD Explorations 2004; 6 (2): 77–86
Goldie SJ, Kim JJ, Kobus K, et al. Cost-effectiveness of HPV 16, 18 vaccination in Brazil. Vaccine 2007; 25 (33): 6257–70
Chung Yin K, Pamela MM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value Health 2009; 12 (4): 521–9
Tappenden P, Chilcott J, Eggington S, et al. Option appraisal of population-based colorectal cancer screening programmes in England. Gut 2007 May 1; 56 (5): 677–84
Wong S. Computational methods in physics and engineering. 2nd ed. Singapore: World Scientific Publishing Co. Pte. Ltd, 1997
Forrester M, Pettitt A, Gibson G. Bayesian inference of hospital-acquired infectious diseases and control measures given imperfect surveillance data. Biostat 2007 Apr 1; 8 (2): 383–401
Goldie SJ, Weinstein MC, Kuntz KM, et al. The costs, clinical benefits, and cost-effectiveness of screening for cervical cancer in HIV-infected women. Ann Intern Med 1999; 130: 97–107
Dayhoff JE, De Leo JM. Artificial neural networks. Cancer 2001; 91 (S8): 1615–35
Tan SYGL, van Oortmarssen GJ, Piersma N. Estimating parameters of a microsimulation model for breast cancer screening using the score function method. Ann Operat Res 2003; 119 (1): 43–61
Vanni T, Legood R, Franco E, et al. Cost-effectiveness of strategies for managing women presenting atypical squamous cells of unknown significance in Brazil. London: London School of Hygiene and Tropical Medicine, 2008
Blower S, Dowlatabadi H. Sensitivity and uncertainty analysis of complex-models of disease transmission–an HIV model, as an example. Int Stat Rev 1994; 62 (2): 229–43
Fylstra D, Lasdon L, Watson J, et al. Design and use of the Microsoft Excel Solver. INTERFACES 1998 Sep 1; 28 (5): 29–55
Lasdon LS, Waren AD, Jain A, et al. Design and testing of a generalized reduced gradient code for nonlinear programming. ACM Trans Math Softw 1978; 4 (1): 34–50
Gill PE, Murray W, Wright MH. Practical optimization. San Diego (CA): Academic Press, 1981
Taylor DCA. Methods of model calibration: a comparative approach. ISPOR 12th Annual International Meeting; 2007 May 18-23; Arlington (VA) [online]. Available from URL: http://www.ispor.org/awards/12meet/MC1-Taylor.pdf [Accessed 2009 Dec 12]
Vanni T, Legood R, White RG. Calibration of disease simulation model using an engineering approach. Value Health 2010; 13 (1): 157
Fryback DG, Stout NK, Rosenberg MA, et al. The Wisconsin breast cancer epidemiology simulation model. J Natl Cancer Inst Monogr 2006; (36): 37–47
Karnon J, Czoski-Murray C, Smith KJ, et al. A hybrid cohort individual sampling natural history model of agerelated macular degeneration: assessing the cost-effectiveness of screening using probabilistic calibration. Med Decis Making 2009 May 1; 29 (3): 304–16
Karnon J, Campbell F, Czoski-Murray C. Model-based cost-effectiveness analysis of interventions aimed at preventing medication error at hospital admission (medicines reconciliation). J Eval Clin Pract 2009; 15 (2): 299–306
Carlton J, Karnon J, Czoski-Murray C, et al. The clinical effectiveness and cost-effectiveness of screening programmes for amblyopia and strabismus in children up to the age of 4–5 years: a systematic review and economic evaluation. Health Technol Assess 2008 Jun; 12 (25): iii, xi-194
Karnon J, Jones R, Czoski-Murray C, et al. Cost-utility analysis of screening high-risk groups for anal cancer. J Public Health 2008; 30 (3): 293–304
Karnon J, McIntosh A, Dean J, et al. A prospective hazard and improvement analytic approach to predicting the effectiveness of medication error interventions. Saf Sci 2007; 45 (4): 523–39
Gelman A, Carlin JB, Stern HS, et al. Bayesian data analysis. 2nd ed. Boca Raton (FL): Chapman and Hall/CRC, 2004
Welton NJ, Ades AE. Estimations of Markov transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis and model calibration. Med Decis Making 2005; 25 (6): 633–45
O’Hagan A, Buck CE, Daneshkhah A, et al. Uncertain judgements: eliciting expert probabilities. Chichester: Wiley, 2006
De Angelis D, Sweeting M, Ades A, et al. An evidence synthesis approach to estimating hepatitis C prevalence in England and Wales. Stat Methods Med Res 2009; 18: 361–79
Welton NJ, Ades AE. A model of toxoplasmosis incidence in the UK: evidence synthesis and consistency of evidence. J R Stat Soc Ser C Appl Stat 2005; 54 (2): 385–404
Goubar A, Ades AE, Angelis DD, et al. Estimates of human immunodeficiency virus prevalence and proportion diagnosed based on Bayesian multiparameter synthesis of surveillance data. J R Stat Soc Ser A Stat Soc 2008; 171 (3): 541–80
Morris S, Devlin N, Parkin D. Economic analysis in health care. 1st ed. Chichester: Wiley, 2007
Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in health and medicine. 1st ed. New York: Oxford University Press, 1996
Briggs A, Sculpher M, Claxton K. Decision modelling for health economic evaluation. 1st ed. Oxford: Oxford University Press, 2007
Acknowledgements
No sources of funding were used to conduct this study or prepare this manuscript. The authors have no conflicts of interest that are directly relevant to the content of this review.
For the invaluable advice provided, the authors thank Michael Pickles and Andrew Cox.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Vanni, T., Karnon, J., Madan, J. et al. Calibrating Models in Economic Evaluation. Pharmacoeconomics 29, 35–49 (2011). https://doi.org/10.2165/11584600-000000000-00000
Published:
Issue Date:
DOI: https://doi.org/10.2165/11584600-000000000-00000