Skip to main content

Advertisement

SpringerLink
Log in

Correction for Measurement Errors in Survey Research: Necessary and Possible

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

Survey research is the most frequently used data collection method in many disciplines. Nearly, everybody agrees that such data contain serious measurement errors. However, only few researchers try to correct for them. If the measurement errors in the variables vary, the comparison of the sizes of effects of these variables on each other will be wrong. If the sizes of the measurement errors are different across countries, cross-national comparisons of relationships between variables cannot be made. There is ample evidence for these differences in measurement errors across variables, methods and countries (Saris and Gallhofer in Design, evaluation and analysis of questionnaires for survey. Wiley, Hoboken, 2007; Oberski in Measurement errors in comparative surveys. PhD thesis, University of Tilburg, 2011). Therefore, correction for measurement errors is essential for the social sciences. The correction for measurement errors can be made in a simple way, but it requires that the sizes of the error variances are known for all observed variables. Many experiments are carried out to determine the quality of questions. The relationship between the quality and the characteristics of the questions has been studied. Because this relationship is rather strong, one can also predict the quality of new questions. A program SQP has been developed to predict the quality of questions. Using this program, the quality of the questions (complement of error variance) can be obtained for nearly all questions measuring subjective concepts. For objective variables, other research needs to be used (e.g., Alwin in Margins of error: a study of reliability in survey measurement. Wiley, Hoboken, 2007). Using these two sources of information, making correction for measurement error in survey research is possible. We illustrate here that correction for measurement errors can and should be performed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. We are grateful to Wiebke Weber of RECSM for collecting the data in Table 1. Pei-Shan Liao of the Academia Sinica and Zih-Wei Wang of the National Taipei University reported to us that in the Taiwanese Sociological Journal and the Political science review, the situation was very similar: Together these journals published in 2010–2012 in total 67 papers of which 18 used survey research and 9 mentioned the problem of measurement errors but only one makes corrections for measurement errors. We are very grateful for this information.

  2. y i  = f i  + e i and cov(f i e i ) = 0 then var(y i ) = var(f i ) + var(e i ) and dividing by var(y i ) gives 1 = [var(f i )/var(y i )] + [var(e i )/var(y i )] or 1 = q 2 i  + var(e i ) ,so the total variance of y i , which is equal to 1, can be decomposed into the quality of the question i plus the proportion error in the variance in the observed variable y i . Normally, q 2 i has been called the reliability of a measurement instrument (Lord and Novick 1968) but we prefer the term quality for reason that will become clear in Sect. 4.

  3. http://essedunet.nsd.uib.no/cms/topics/measurement/.

  4. However, this information is not enough to correct for measurement errors in all ESS questions because at the same time in the ESS more than 60,000 questions were asked with respect to values, opinions, attitudes, preferences, feelings, etc. This illustrates that a different approach is required.

  5. SQP is free of change available at sqp.upf.edu.

  6. It can be shown that the CMV = r 1j m ij m 2j r 2j .

References

  • Alwin, D. F. (2007). Margins of error: A study of reliability in survey measurement. Hoboken: Wiley.

    Book  Google Scholar 

  • Alwin, D. F., & Krosnick, I. A. (1991). The reliability of survey attitude measurement. The influence of question and respondent attributes. Sociological Methods and Research, 20, 139–181.

    Article  Google Scholar 

  • Andrews, F. M. (1984). Construct validity and error components of survey measures: A structural equation approach. Public Opinion Quarterly, 48, 409–442.

    Article  Google Scholar 

  • Belson, W. (1981). The design and understanding of survey questions. London: Gower.

    Google Scholar 

  • Biemer, P. R. (2011). Latent class analysis of survey errors. Hoboken: Wiley.

    Google Scholar 

  • Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.

    Article  Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrices. Psychological Bulletin, 56, 81–105.

    Article  Google Scholar 

  • De Castellarnau, A., & Saris, W. E. (2014). A simple way to correct for measurement errors. European Social Survey Education Net (ESS EduNet). http://essedunet.nsd.uib.no/cms/topics/measurement/

  • Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. Hoboken: Wiley.

    Google Scholar 

  • Goldberger, A. S., & Duncan, O. D. (Eds.). (1973). Structural equation models in the social sciences. New York: Seminar Press.

    Google Scholar 

  • Hagenaars, J. (1988). Latent structure model with direct effects between indicators; local dependency models. Sociological Methods and Research, 16, 379–405.

    Article  Google Scholar 

  • Hambleto, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. London: Sage.

    Google Scholar 

  • Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago: University of Chicago Press.

    Google Scholar 

  • Heise, D. R. (1969). Separating reliability and stability in test–retest-correlation. American Sociological Review, 34, 93–101.

    Article  Google Scholar 

  • Jöreskog, K. G. (1973). A general method for estimating a linear structural equation system. In A. S. Goldberger & O. D. Duncan (Eds.), Structural equation models in the social sciences. New York: Academic Press.

    Google Scholar 

  • Költringer, R. (1993). Gültigkeit von Umfragendaten. Wien: Bohlau.

    Google Scholar 

  • Költringer, R. (1995). Measurement quality in Austrian personal interview surveys. In W. E. Saris & A. Münnich (Eds.), The multitrait-multimethod approach to evaluate measurement instruments (pp. 207–225). Budapest: Eötvös University Press.

    Google Scholar 

  • Lawley, D. N., & Maxwell, A. E. (1971). Factor analysis as a statistical method. London: Butterworth.

    Google Scholar 

  • Lord, F., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesly.

    Google Scholar 

  • Mokken, R. J. (1971). A theory and procedure of scale analysis: With applications in political research. The Hague: Mouton.

    Book  Google Scholar 

  • Molenaar, N. I. (1986). Formuleringseffecten in survey-interviews. PhD thesis, Amsterdam: Free University.

  • Oberski, D.T. (2011). Measurement errors in comparative surveys. PhD thesis, University of Tilburg.

  • Oberski, D.T., Gruner, T., & Saris, W. E. (2011). The prediction procedure the quality of the questions based on the present data base of questions In W. E Saris, D. Oberski, M. Revilla, D. Zavalla, L. Lilleoja, I. Gallhofer, & T. Grüner. (Eds.), The development of the Program SQP 2.0 for the prediction of the quality of survey questions. RECSM Working paper 24, chapter 6.

  • Saris, W. E, Oberski, D., Revilla, M., Zavalla, D., Lilleoja, L., Gallhofer, I., & Grüner, T. (2011). The development of the Program SQP 2.0 for the prediction of the quality of survey questions. RECSM Working paper 24.

  • Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Education Research.

    Google Scholar 

  • Saris, W. E., & Gallhofer, I. N. (2007). Design, evaluation and analysis of questionnaires for survey research. Hoboken: Wiley.

    Book  Google Scholar 

  • Saris, W. E., & Gallhofer, I. N. (2014). Design, evaluation and analysis of questionnaires for survey research (2nd ed.). Hoboken: Wiley.

    Book  Google Scholar 

  • Saris, W. E., Satorra, A., & Coenders, G. (2004). A new approach for evaluating quality of measurement instruments. Sociological Methodology, 3, 311–347.

    Article  Google Scholar 

  • Oberski, D. T., & Satorra, A. (2013). Measurement error models with uncertainty about the error variance. Structural Equation Modeling, 20, 409–428.

  • Scherpenzeel, A. C. (1995). A question of quality. Evaluating survey questions by multitrait-multimethod studies. Leidschendam: KPN Research.

    Google Scholar 

  • Schuman, H., & Presser, S. (1981). Questions and answers in attitude survey: Experiments on question form, wording and context. New York: Academic Press.

    Google Scholar 

  • Sudman, S., & Bradburn, N. M. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey Bass.

    Google Scholar 

  • Torgerson, W. S. (1958). Theory and methods of scaling. London: Wiley.

    Google Scholar 

  • Tourangeau, R., Rips, J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, MA: Cambridge University Press.

    Book  Google Scholar 

  • Van Schuur, W. H. (1997). Nonparametric IRT models for dominance and proximity data. In M. Wilson, G. Engelhard Jr, & K. Draney (Eds.), Objective measurement: Theory into practice (Vol. 4, pp. 313–331). Greenwich: Ablex Publishing Corporation.

    Google Scholar 

  • Vermunt, J. K. (2003). Multilevel latent class models. Sociological Methodology, 33, 213–239.

    Article  Google Scholar 

  • Wiley, D. E., & Wiley, I. A. (1970). The estimation of measurement error in panel data. American Sociological Review, 35, 112–117.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. RECSM, Universitat Pompeu Fabra, Parc de Recerca, Despatx 24.406, Ramón Trías Fargas, 25-27, 08005, Barcelona, Spain

    Willem E. Saris & Melanie Revilla

Authors
  1. Willem E. Saris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melanie Revilla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Willem E. Saris.

Appendices

Appendix 1

See Table 7.

Table 7 Procedure to correct for measurement error using LISREL
Full size table

Appendix 2: The different questions

figure a

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saris, W.E., Revilla, M. Correction for Measurement Errors in Survey Research: Necessary and Possible. Soc Indic Res 127, 1005–1020 (2016). https://doi.org/10.1007/s11205-015-1002-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-015-1002-x

Keywords

Search

Navigation