PROMIS Series
Representativeness of the Patient-Reported Outcomes Measurement Information System Internet panel

https://doi.org/10.1016/j.jclinepi.2009.11.021Get rights and content

Abstract

Objectives

To evaluate the Patient-Reported Outcomes Measurement Information System (PROMIS), which collected data from an Internet polling panel, and to compare PROMIS with national norms.

Study Design and Setting

We compared demographics and self-rated health of the PROMIS general Internet sample (N = 11,796) and one of its subsamples (n = 2,196) selected to approximate the joint distribution of demographics from the 2000 U.S. Census, with three national surveys and U.S. Census data. The comparisons were conducted using equivalence testing with weights created for PROMIS by raking.

Results

The weighted PROMIS population and subsample had similar demographics compared with the 2000 U.S. Census, except that the subsample had a higher percentage of people with higher education than high school. Equivalence testing shows similarity between PROMIS general population and national norms with regard to body mass index, EQ-5D health index (EuroQol group defined descriptive system of health-related quality of life states consisting of five dimensions including mobility, self-care, usual activities, pain/discomfort, anxiety/depression), and self-rating of general health.

Conclusion

Self-rated health of the PROMIS general population is similar to that of existing samples from the general U.S. population. The weighted PROMIS general population is more comparable to national norms than the unweighted population with regard to subject characteristics. The findings suggest that the representativeness of the Internet data is comparable to those from probability-based general population samples.

Introduction

What is new?

  • Date collected from an Internet polling panel can be effectively weighted by raking method.

  • Equivalence testing can be used to test equivalence between two groups of data such as the internet collected data and national norms or census data.

  • The representativeness of the Internet data can be comparable to those from probability-based general population samples.

There are many methods for collecting survey data, such as face-to-face or telephone interviews, mail, fax, e-mail, or Web-based surveys [1]. The number of individuals who have access to the Internet is growing exponentially, and the population of Internet users from which general surveys might sample is increasing [2]. As a result, the number of studies using Internet data collection (IDC) has increased, presenting new opportunities and challenges in data collection and analyses.

Limitations of traditional random digit dialing (RDD) with regard to obtaining representative samples have further stimulated IDC. These limitations have increased because of widespread screening of incoming calls and the increasing number of cell phone users without home phone “landlines.” Nonresponse associated with RDD sampling is higher than that of personal interviews, and it is possibly less appropriate for personal or sensitive questions, if there is no prior contact [3]. Compared with conventional data methods, such as paper survey and face-to-face or phone interviews, there are several noteworthy advantages to IDC: it is cost-effective to study large and heterogeneous samples; it has the ability to recruit specialized samples (e.g., people with rare characteristics); and the standardization of data collection process makes studies easy to replicate. However, IDC also has disadvantages, such as difficulty ensuring the integrity, security, reliability, and validity of data collected [2], [4], [5]; higher rates of loss of follow-up [6]; and biases in the population that often accesses the Web, despite not being geographically restricted [2].

A high response rate is commonly taken as an indicator of survey validity [7]. In addition, selection bias is an important consideration because of its impact on generalizability [8]. Some studies have shown that IDC led to a significantly lower response rate than traditional mailed surveys [9] or found significant differences in the sample characteristics and overall costs between telephone and Web surveys used to collect data on the corporate reputation of an international firm [10]. In contrast, other studies have found IDC to produce similar reliability and validity as traditional collection methods [11], [12], [13], [14], [15], [16], [17]. Schillewaert et al. [18] compared respondents recruited by postal mail, telephone, Internet panels, and pop-up Internet surveys, and found that online and offline methods yielded respondents with similar attitudes, interests, and opinions after controlling for sociodemographics from census data.

Substantial data collection efficiency, low cost, and widespread availability of Internet access among diverse groups are stimulating increased usage of Web-based surveys [10]. However, Internet surveys may not be representative of a population of interest, because the subpopulation with access may be atypical. Weighting adjustments can be applied to surveys to compensate for nonresponse, noncoverage, unequal selection probability, and sampling fluctuation from known population values.

Different weighting methods have been developed, such as cell weighting and raking [19]. The purpose of weighting adjustments is to make the weighted sample distributions conform to distributions or estimates from an external source or a large high-quality survey. For each of the different weighting methods, two weighting approaches can be used: population weighting and sample weighting. When population-weighting adjustments are used, the respondent sample is weighted so that the weighted sample distribution is the same as the distribution of the population across classes (such as population estimates by age and sex). Sample-weighting adjustments weight respondents within classes so that the profile of respondents across classes is equivalent to the profile of the entire survey sample [19], [20].

The cell-weighting method adjusts the sample weights so that the sample distributions or totals conform to the population distributions or totals on a cell-by-cell basis. The assumption underlying cell-weighting adjustment for nonresponse is that the respondents within a given cell represent the nonrespondents within that cell, which implies that data are missing at random [21]. A practical limitation of cell weighting is that as the number of stratification variables and the number of cells increase, the number of subjects in each cell decreases, thus producing less-stable aggregated estimates.

Raking matches cell counts with the marginal distributions of the grouping variables used in the weighting scheme [19], [21], [22]. Raking is an iterative proportion procedure, which performs cell-by-cell adjustments over the various univariate distributions to make the weighted sample cells match external values, such as the U.S. Census data. This process is repeated iteratively until there is convergence between the weighted sample and the external distributions [23].

Propensity score adjustment can alleviate the confounding effects of the selection mechanism in observational studies by achieving a balance of covariates between comparisons [24], [25]. Harris Interactive (http://www.harrisinteractive.com/ developed software for performing propensity score weighting (PSW) to correct for attitudinal and behavioral differences typically found in online respondents [26]. Propensity score matching [24], on which PSW is based, has been used to ensure that comparison groups have similar characteristics when random assignment is not possible. Schonlau and Van Soest [27] found that the propensity adjustment to correct selection bias in Internet surveys works well for many but not all variables investigated and cautioned against the common practice of using only a few basic variables to correct for selectivity in convenience samples drawn over the Internet.

The Patient-Reported Outcomes Measurement Information System (PROMIS) project aims to develop highly reliable and valid item banks to measure patient-reported symptoms and other aspects of health-related quality of life for administration to persons with a wide range of chronic diseases and demographic characteristics. PROMIS collected data using a polling panel consisting of more than 1 million members who had previously indicated a willingness to respond to online surveys. In this study, we evaluated the distributional characteristics obtained from those who accepted the invitation to complete a survey, created a weighting scheme to compensate for nonresponse and noncoverage to make weighted sample estimates conform to the U.S. population, and generated a subsample through disproportionate sampling to simulate the distribution of the U.S. general population demographics. We compared the PROMIS Internet samples with three U.S. national surveys, and general population with regard to participant demographics, general health, body mass index (BMI), and EQ-5D health index score. Based on these comparisons, inferences were made about the quality and generalizability of the PROMIS Internet sample.

Section snippets

Patient-Reported Outcomes Measurement Information System

PROMIS is a National Institutes of Health Roadmap project that uses item-response theory and computer adaptive testing to provide an accurate, efficient, and publicly accessible system that can be used by medical researchers and health professionals to assess patient-reported outcomes across a number of measurement domains [28]. Five primary domains were selected for initial item-bank development: physical functioning, pain, fatigue, emotional distress, and social-role participation. For the

Results

Demographics of PROMIS general population along with the U.S. Census data are shown in Table 1. For U.S Census data, the mean age was 45 years (standard deviation [SD] = 18), and 48% were males. Most of the participants were whites (74%) followed by Hispanics (11%), blacks (11%), and other races (4%). Fifty-one percent had more than high school education, 29% had high school diploma or equivalent, and the rest (20%) had less than high school education. Fifty-seven percent were married, 23% were

Discussion

With the rapid growth in the use of the Internet in the past decade, the number of studies using IDC has increased significantly. The traditional RDD has limitations with regard to obtaining representative samples because of the increasing number of cell phone users without home phone landlines and widespread screening of incoming calls. Nonresponse rate associated with RDD sampling is higher than that for personal interviews [3]. To overcome these limitations, PROMIS collected

Acknowledgment

The authors wish to thank Victor Gonzalez for his technical assistance in the preparation of this article.

References (44)

  • U. Reips

    The Web experiment method: advantages, disadvantages, and solutions

  • A.T. Nathanson et al.

    Windsurfing injuries: results of a paper- and Internet-based survey

    Wilderness Environ Med

    (1999)
  • J.C. Wyatt

    When to use web-based surveys

    J Am Med Inform Assoc

    (2000)
  • W.C. Schmidt

    World-Wide Web survey research: benefits, potential problems, and solutions

    Behav Res Meth Instrum Comput

    (1997)
  • H. Bell et al.

    Can you ask that over the telephone? Conducting sensitive and controversial research using random-digit dialing

    Med Law

    (2006)
  • O.L. Strickland et al.

    Measurement issues related to data collection on the World Wide Web

    Adv Nurs Sci

    (2003)
  • M.H. Birnbaum

    Human research and data collection via the Internet

    Annu Rev Psychol

    (2004)
  • M. Schonlau

    Will Web surveys ever become part of mainstream research?

    J Med Internet Res

    (2004)
  • G. Eysenbach et al.

    Using the Internet for surveys and health research

    J Med Internet Res

    (2002)
  • P. Leece et al.

    Internet versus mailed questionnaires: a randomized comparison (2)

    J Med Internet Res

    (2004)
  • C.A. Roster et al.

    A comparison of response characteristics from Web and telephone surveys

    Int J Market Res

    (2004)
  • T. Buchanan et al.

    Research on the Internet: validation of a World-Wide Web mediated personality scale

    Behav Res Meth Instrum Comput

    (1999)
  • C. Senior et al.

    An investigation into the perception of dominance from schematic faces: a study using the World-Wide Web

    Behav Res Meth Instrum Comput

    (1999)
  • J.H. Krantz et al.

    Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness

    Behav Res Meth Instrum Comput

    (1997)
  • P. Ritter et al.

    Internet versus mailed questionnaires: a randomized comparison

    J Med Internet Res

    (2004)
  • R.N. Davis

    Web-based administration of a personality questionnaire: comparison with traditional methods

    Behav Res Meth Instrum Comput

    (1999)
  • H. Raat et al.

    Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): Internet administration compared with the standard paper version

    Qual Life Res

    (2007)
  • N. Schillewaert et al.

    Comparing response distributions of offline and online data collection methods

    Int J Market Res

    (2005)
  • G. Kalton et al.

    Weighting methods

    J Official Stat

    (2003)
  • G. Kalton et al.

    The treatment of missing survey data

    Survey Methodol

    (1986)
  • R.J.A. Little et al.

    Statistical analysis with missing data

    (2002)
  • R.J.A. Little et al.

    Models for contingency tables with known margins when target and sampled populations differ

    J Am Stat Assoc

    (1991)
  • Cited by (0)

    This work was funded by the National Institutes of Health (NIH) through the NIH Roadmap for Medical Research Cooperative Agreement (1U01-AR052177).

    View full text