FormalPara Key Points for Decision Makers

Patient-reported outcome (PRO) endpoints are often necessary to adequately evaluate the treatment benefit provided by new medical products (e.g., drugs) in clinical trials.

The movement from paper-based to electronic PRO (ePRO) data collection has enhanced the integrity and accuracy of PRO data in clinical trials.

The US Food and Drug Administration has made it clear that electronic capture of clinical trial source data is preferred over paper-based data collection.

The bring your own device (BYOD) approach to ePRO data collection in clinical trials has considerable promise, but a number of questions need to be answered prior to its routine adoption.

1 Introduction

In the evaluation of new medical products, regulatory agencies (e.g., the US Food and Drug Administration [FDA]) rely on clinical trial endpoints that are believed to adequately assess the treatment benefit provided by the product. Treatment benefit is demonstrated by evidence that the treatment has a positive impact on how long a patient lives or how a patient feels or functions in daily life [1]. The regulatory uses for documented treatment benefit are marketing approval and label claims [2]. A claim of treatment benefit must be supported by substantial evidence from adequate and well controlled studies using well defined and reliable assessments [3].

Studies to assess treatment benefit are usually comparative efficacy trials, but can be comparative safety trials (e.g., assessing relative levels of a specific treatment-related toxicity). Along with biomarkers (e.g., HbA1C, blood pressure) and survival, clinical outcome assessments (COAs) can be used as primary or secondary endpoints in trials to determine whether a drug provides a treatment benefit. COAs include patient-reported outcomes (PROs), clinician-reported outcomes, observer-reported outcomes, and performance outcomes [4]. The focus of this article is the assessment of PRO endpoints in clinical trials.

A PRO instrument collects treatment benefit data directly from patients, without interpretation by clinicians or others [5]. As recognized by the FDA in its Guidance for Industry titled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (aka ‘PRO Guidance’), PRO instruments should be used when measuring concepts best known by the patient or best measured from the patient’s perspective [5]. The purpose of this article is to describe electronic PRO (ePRO) data collection in clinical trials assessing the efficacy of new medical products for regulatory approval and to provide a brief overview of the current technology and emerging trends.

2 The Shift Toward ePRO Data Collection

Based on the evidence that has been mounting over the past 25 years, unsupervised paper-based self-reports of function and symptom experience are far from optimal compared with data collected electronically [6–8]. For example, Stone and colleagues demonstrated that, although study subjects reported high compliance (i.e., daily data entry at required times) with a paper diary, their actual compliance (i.e., actual completion times in relation to when diaries should have been completed) was very low when compared with actual compliance of subjects using an electronic diary [7]. As stated by Ganser and colleagues [9, p. 49], paper-based approaches to patient-reported data collection can “result in untimely, unreadable, missing, illogical or otherwise faulty data.” Hence, a number of scientific, regulatory, and practical considerations are driving the shift to electronic capture of PRO endpoints in clinical trials. The emergence of ePRO technologies has been a profoundly significant innovation for PRO endpoint assessment; it has significantly increased data collection capabilities and options in clinical trials.

ePRO systems can lead to more accurate and complete data [7, 9], improved protocol compliance [10], avoidance of secondary data entry errors [9], easier implementation of skip patterns [11], less administrative burden [10, 12], high respondent acceptance [12–14], reduced sample size requirements [9], and potential cost savings [15, 16]. Compliance with self-reported data entry requirements specified by the protocol in a clinical trial can be improved through the use of alarms and reminders in addition to more timely follow-up with non-compliant subjects that is enabled by nearly real-time investigator data monitoring capabilities [17]. However, it must be stated that the many advantages of ePRO data capture may not materialize if study subjects do not receive adequate training on the ePRO device/system and the data entry requirements of the study protocol.

The accelerating movement from paper diaries and questionnaires to ePRO data collection systems has enhanced the integrity and accuracy of data collected in clinical trials [9]. This transition, along with FDA’s release of the PRO Guidance, has elevated the science of PRO measurement. Among the regulatory bodies of the world, the FDA has taken a leadership role in advancing sound COA endpoint assessment. For example, the following quote from the PRO Guidance explicitly endorses the functionality of electronic data collection platforms and, hence, implicitly endorses the shift to ePRO: “If a patient diary or some other form of unsupervised data entry is used, we plan to review the clinical trial protocol to determine what steps are taken to ensure that patients make entries according to the clinical trial design and not, for example, just before a clinic visit when their reports will be collected” [5, p 9].

Electronic data collection provides a record of the date and time entries are made and can impose specified time windows for data entry [18]. This functionality can help prevent (or identify) back-filling or forward-filling of entries that can undermine the quality and accuracy of paper-based diaries or questionnaires [7]. In addition, ePRO systems can decrease missing or unusable data by compelling a response and by not allowing entry of out-of-range, contradictory, or extraneous responses. The FDA Guidance for Industry titled Electronic Source Data in Clinical Investigations provides further support for the electronic capture of PRO endpoints in clinical trials [19].

3 Ensuring High-Quality Measurement

The advantages of electronic data capture are well documented, but those advantages mean little if measurement error is introduced by poor implementation of a PRO instrument on an ePRO platform. This is a particularly important consideration when, as is often the case, the PRO instrument being used was initially developed as a paper-based questionnaire. The migration of paper-based instruments to electronic data collection modes must be properly planned and executed so that respondents interpret and respond to the PRO instrument’s items the same way regardless of the data collection mode. To address this issue, the International Society of Pharmacoeconomics and Outcomes Research (ISPOR) released a task force report in 2009 that provided recommendations as to the type and amount of evidence needed to support measurement equivalence between electronic and paper-based PRO measures [20]. Subsequently, another ISPOR task force report expanded those recommendations to situations where more than one mode of PRO data collection is being considered for a clinical trial [21]. This task force recognized that sufficient evidence may be available in the future to support the assumption of measurement equivalence across PRO data collection modes in most circumstances where a faithful migration has occurred [21], since a growing body of empirical research shows that, as long as substantive content changes are not made, paper and electronic versions of PRO instruments generally provide comparable data [22].

In addition, the ePRO Consortium, which has ten ePRO system providers as members, undertook the development of best practice recommendations to facilitate the successful implementation of ePRO data capture. The ePRO Consortium (http://c-path.org/programs/epro) was established in 2011 by the non-profit Critical Path Institute to collaboratively address scientific, not commercial, issues related to the electronic capture of PRO endpoints in clinical trials. The best practice documents (http://c-path.org/programs/epro/#section-5648) completed by the ePRO Consortium and available through its Web page include Best Practices for Migrating Existing Patient-Reported Outcome Instruments to a New Data Collection Mode, Best Practices for Electronic Implementation of Patient-Reported Outcome Response Scale Options, and Best Practices for Maximizing Electronic Data Capture Options during the Development of New Patient-Reported Outcome Instruments. The latter document addresses the importance of considering all possible data collection modes when developing new instruments. Too many PRO instrument developers are paper-centric and fail to contemplate alternative data collection modes when developing their items and selecting response scales. For example, a true visual analog scale (VAS) has descriptive verbal anchors at each end with an uninterrupted line in between. On a self-administered paper-based questionnaire, the respondent can be asked to place a mark or ‘X’ at the point on the line that best reflects his or her response; however, for a telephone-administered version of the same questionnaire, there is no way to operationalize the VAS response options without converting it to a numeric rating scale.

Another consideration that can undermine high-quality measurement in clinical trials that transcends individual instrument issues is the development and deployment of the overall ePRO system; this is far from a trivial task. ISPOR’s ePRO Systems Validation Task Force developed recommendations for clinical trial teams regarding the validation of electronic systems used to collect PRO data in clinical trials [23]. The task force’s primary goal was to bring attention to the technical nature of ePRO systems and the essential steps in the ePRO system validation process. It is quite clear that ePRO system validation requires collaboration between the ePRO system provider and the clinical trial team, since both parties have a substantive and complementary role to play in the ePRO system validation process.

4 ePRO Data Collection Technologies

The availability of new data capture technologies has enabled clinical trial protocols to be designed in ways that allow for more timely and patient-friendly approaches to PRO endpoint assessment [21, 24]. The selection of an ePRO data collection technology should never be an afterthought in a drug development program. There are a number of factors that must be considered when selecting the most appropriate mode of data collection, including patient population, location of data collection, characteristics of the instrument, data collection schedule, feasibility, and cost. Eremenco and colleagues [21] provide an overview of these factors. Selecting the most appropriate approach to ePRO data collection is essential to the success of the clinical trial.

In clinical research, PRO data collection can occur at a study site (e.g., clinic) or off-site in an unsupervised setting (e.g., subject’s home, workplace, or school). To enable more timely and accurate self-reporting and to avoid potential recall bias, off-site assessment of symptom or functional endpoints is often optimal. For either study site or off-site (i.e., remote) PRO data collection, ePRO platforms are recommended over paper-based approaches, but it is especially critical for unsupervised, remote data entry due to the reasons discussed above.

A primary distinction in the types of ePRO platforms is between telephone-based interactive voice response (IVR) systems and screen-based systems. Within the screen-based approaches are device-based systems (i.e., proprietary software installed on a specific hardware platform), Web-based systems, and downloadable application (app)-based systems that utilize the subject’s or site’s screen-based device. The different types of available ePRO data collection technologies are described briefly below; further description of these technologies is provided by Eremenco and colleagues [21]. Since using downloadable apps for collecting PRO data in clinical trials is an emerging methodology, this topic is only mentioned briefly in the descriptions below, but will be covered in more depth in the context of bring your own device (BYOD) trials later in this article.

4.1 Interactive Voice Response Systems

IVR systems leverage landlines or cellular telephones already available to subjects; hence, no hardware deployment is required and minimal subject training is necessary. IVR systems use a pre-recorded question and response option script and subjects respond via the phone’s keypad. Data are stored directly to a central database. IVR systems can be implemented to call the subject or to allow the subject to call the system at scheduled data collection time points.

4.2 Handheld Devices

Although self-reported clinical trial data had been collected via computerized systems previously, the adoption of personal digital assistants (PDAs) in the late 1980s and early 1990s truly changed the PRO landscape and accelerated the movement to ePRO data capture [11, 25]. Initially, these devices used offline synchronization and required temporary data storage on the device until the data could be uploaded to a local (e.g., site-based) computer or central server [17]. Advances involving the integration of wireless and cellular phone capabilities expanded their utility exponentially since they allowed data to be transferred immediately from the device to a central server/database. Hence, most of these handheld devices are now mobile telephones (e.g., smartphones) and they have become the mainstay of ePRO data collection. Their portability and touch screen functionality provide a significant advantage over other data collection modes, particularly for clinical trials requiring frequent remote PRO data capture (e.g., symptom or event-based diaries).

Although most handheld devices used in clinical trials are device-based systems, handhelds can also provide access to Web-based data collection portals (e.g., interactive Web response system) or provide ePRO functionality through a downloadable app. However, it must be noted that using a handheld device to access a website for clinical trial data collection assumes that functionality and rendering of the PRO instrument’s items on the device’s screen are optimized for that particular operating system, browser, and screen size [17].

4.3 Tablet Computers

Tablet computers represent another relatively mobile, touch screen-enabled data collection platform. Although the size differences are narrowing, tablets are generally larger than traditional handheld devices and smaller than laptop computers. The larger screen size of a tablet compared with a handheld, and the relative mobility and ease of use compared with a laptop, make tablets the primary platform for site-based ePRO data collection. As with handheld devices, tablets can be device-based systems, provide access to Web-based portals, or be used with downloadable ePRO apps.

4.4 Desktop or Laptop Computers

Although no longer used as often in clinical trials for data entry by patients, desktop and laptop computers remain platforms for collecting PRO data. They generally have larger screens but usually lack touch screen functionality; subjects use a keyboard and/or mouse to enter responses. Handheld devices and tablets have become the predominant ePRO platforms, but desktop and laptop computers are still used to capture site-based PRO data as well as remotely captured data. The larger screen size of a desktop or laptop can be an advantage in certain patient populations or in situations where a larger representation of a PRO instrument or its items/components would be preferable. As with the other Internet-enabled screen-based platforms, desktop and laptop computers can be device-based systems, provide access to Web-based portals, or be used with downloadable ePRO apps.

4.5 Digital Pen

An ePRO platform that doesn’t fit into either a screen-based or telephone-based system category is the digital pen. Although digital pens are not used to collect a significant share of ePRO data in clinical trials, they are available. With digital pen technology, study subjects write their responses on a paper questionnaire that has been specially printed to enable the pen to locate its position on the questionnaire and record what is being written. Along with a ballpoint ink cartridge, digital pens contain a pressure sensor, a camera to capture the written data, a microprocessor, memory, and a battery. Some digital pens have a Bluetooth transceiver that enables the responses captured by the pen’s camera to be uploaded to a Web-based server in real time. Other pens use a docking cradle attached to a computer that uploads the data to the server via the Internet.

5 The Emergence of Bring Your Own Device (BYOD)

As mentioned above, handheld devices have become the mainstay for remote (i.e., off-site, unsupervised) ePRO data collection in clinical trials. The conventional approach has been to provide to study subjects (via the study sites) a handheld device with a device-based proprietary software program (i.e., a stand-alone device). This approach has worked well in ensuring subjects have a dedicated, ‘locked down’ device on which they can enter study data in a controlled manner; however, as described by Yeomans [17], there are a number of important considerations with this approach, including the following: provisioning costs (i.e., costs associated with the purchase or lease of the devices used in the trial); supply issues (i.e., delivering the devices to the study sites for distribution to the subjects and collection of the devices at study completion); training of the site staff so that they can train subjects on the handling and use of the device; maintenance (i.e., replacement of faulty devices); and maintaining a help desk for device-related questions from sites or subjects.

Hence, this can be costly for sponsors, challenging for study sites, and inconvenient for study subjects, who may already carry a separate handheld device for personal use. Due to the widespread use of personal smartphones, leveraging study subjects’ own mobile device for remote PRO data collection is an attractive alternative for clinical trials. One way this can be accomplished is by having study subjects access Web-based data collection portals from their own Internet-enabled mobile device. Yeomans [17] briefly introduces three examples of efficacy trials for drugs or devices that incorporated this type of Web-based PRO data collection. The other option involves having the subject download a software app to their own mobile device. The subject runs the app locally on the device and completes the PRO instrument. The data are then transmitted via the Internet (Wi-Fi) or cellular signal. We know of no publicly available information on clinical trials that have used an app to collect primary or secondary PRO-based efficacy data. However, there is increasing interest in developing PRO data collection apps and, as a result, it is imperative that both the potential benefits and challenges be delineated and seriously considered. Although this approach has many strengths and addresses some shortcomings of current approaches, there are also scientific and operational questions that must be addressed.

6 Potential Benefits and Challenges of BYOD

In the field of medical research, BYOD, in its broadest definition, means allowing participants in a clinical trial to use their own computer devices (e.g., smartphone, tablet, laptop, desktop PC) to access and respond to study-related PRO questionnaires. As discussed, this is a departure from traditional studies utilizing ePRO where patients are provided with the hardware they need to enter study data, typically a tablet or desktop computer for site-based questionnaires and a PDA or smartphone for remote questionnaire completion. There are a number of factors that make BYOD a compelling model for clinical trials (Box 1); however, there are key outstanding issues that need to be adequately addressed before BYOD becomes a viable model for clinical trials. Although the technology is available to do it today, enthusiasm for doing it should not trump the need to approach it in a measured and thoughtful manner.

figure a

6.1 Downloading Software to Devices

One of the key issues standing in the way of widespread adoption of BYOD in clinical trials is how one gives the study subject access to the PRO instruments they are required to complete during the study. With Web-enabled devices such as smartphones, subjects could access the instruments on their Web browser. However, this approach requires active Internet access in order to view the PRO instruments and enter data. The process of entering data on a mobile device via the Internet can vary based on the subject’s network connection; subjects using their mobile device to enter data on a website may have to wait for the browser screens to load and respond. In addition, not all browsers are created equally and functionality can vary between different browsers, and between different versions of the same browser. Security is a heightened concern when transmitting data over an open Internet connection.

An alternative to entering data on a website is the use of apps, where the subject downloads a small piece of software onto their smartphone that would display the PRO instruments(s) in a relatively consistent manner across handsets, barring differences in screen size. This has the important advantage of allowing subjects to respond to questionnaires and provide PRO data when they do not have Internet access. While native apps can address some of the issues above, they present unique considerations. New functionality for a Web browser can be built and deployed quickly; a native app will take longer, since it must go through an entire software development lifecycle.

However with both of these approaches, Web and app, there can be an issue with ensuring the wide range of devices, operating systems, and Web browsers available are compatible with any app and Web system that is developed. This can be overcome by creating the app and Web system on the widely used operating systems such as Android and iOS, and the widely used Web browsers such as Internet Explorer, Firefox, Safari, and Chrome, to ensure greater access and compatibility. However, this does risk creating issues in countries and populations where other operating systems and browsers are more predominant.

6.2 Equivalence Across Data Collection Modes

Another key issue is that of equivalence of PRO instruments across different data collection platforms or modes, particularly when they are being used to support primary or key secondary endpoints. The FDA expects the sponsor to demonstrate that a PRO instrument that has been ‘modified’ is capturing equivalent data regardless of data collection mode [5]. Based on current recommendations [20, 21], using mixed modes in a clinical trial would require a quantitative measurement equivalence study if there are more than minimal differences in the presentation of the PRO instrument on the different modes (e.g., smartphone screen vs the large monitor of a desktop PC). This approach becomes impractical in the BYOD model where the number of potential devices, operating systems, and browsers available could be extensive. However, with appropriate programming across the various operating systems and browsers, the only material difference becomes the size of the screen (and font) on the device the subject is using to enter data. In this case, an assessment of the measurement equivalence of the smallest screen size to the largest screen that would realistically be available for each operating system should be sufficient. As mentioned previously in regard to ePRO more generally, an accumulation of evidence from BYOD-based studies may, at some point in the future, support the assumption of measurement equivalence across screen-based devices in most circumstances where a faithful migration has been rendered.

6.3 Paying for Data

During traditional ePRO studies where clinical trial subjects are provided hardware by the trial sponsor, data transmission costs (e.g., submitting data from a daily diary completed on a smartphone) are automatically covered by the sponsor with the included SIM card on the device. However, in the BYOD model, the expectation is that the subject is using his or her own device and, as a result, will incur data transmission costs as part of, or in addition to, the subjects’ cellular plan. The subject should have a reasonable expectation of being reimbursed for these costs in addition to any other compensation received for participating in the study.

6.4 People Without Access to a Suitable Device

The underlying assumption of the BYOD model is that study subjects have their own device, which they can, in fact, bring. In FDA parlance, the subjects are required to provide a device that is ‘fit for purpose.’ However, even high global Internet and mobile phone penetration does not guarantee that the patient population for a particular study will all have a suitable device that can access the study measures, and there is little reason to believe that smartphone penetration will ever reach 100 %. For example, in the US ~45 % of all adults own smartphones, but ownership varies widely based on age, income, education, and geographic location. However, Blacks (47 %) and Hispanics (49 %), who are traditionally underrepresented in clinical trials in the US, had higher rates of smartphone ownership than non-Hispanic whites (42 %) [26].

There are two potential approaches to addressing the device access issue:

  • Require ownership of an appropriate device in the inclusion criteria. This would help to ensure that participants in the study have an appropriate device on which to enter data. However, this approach raises serious concerns about selection bias within the sample. Potentially, this could lead to exclusion of important segments of the target population from the clinical trial.

  • Provide participants who do not have appropriate devices with stand-alone hardware, while allowing those with an appropriate device to follow the BYOD model. This approach would avoid issues around excluding certain segments of the target population, but attenuates one of the key perceived strengths of the BYOD model (i.e., avoiding costs associated with providing devices).

Whether selection bias is a problem will depend on the size of the target patient population. If the target population is hard to find, then the idea of providing devices would be attractive as it will allow those that have the right device to use their own, but it would not exclude those that do not. If the target population is easy to attain, then the ownership of the correct device can be used as an inclusion criterion.

6.5 Security, Both Physical and Electronic

When evaluating BYOD solutions, ensuring protection of the subject’s private information and the data being collected should be a paramount concern.

Box 2 provides examples of questions that should be considered when evaluating any BYOD system. The answers to these questions and others like them will impact the level of confidence with BYOD in general and specific solutions in particular.

figure b

Smartphones and tablets present problems when lost or stolen. The physical security can be breached with a brute force attack, so extra design precautions need to be taken to ensure that the patient’s privacy and anonymity are protected. There are also additional security considerations that need to be addressed to make sure that authentication controls are not bypassed. Such privacy and authentication controls are part of a ‘normal’ ePRO build, no matter what modality is being used, and apply to all aspects of a person’s digital life, not just his or her health-related data.

6.6 Ownership

In BYOD studies, the sponsor no longer owns and controls the device in the same way as if the device were deployed to the subjects. It should be recognized that these are multi-purpose devices that cannot be locked down in a way that disables any of the other device’s functions during data collection. In addition to completing the study protocol activities (i.e., responding to the PRO instrument at the appropriate time) on their devices, study subjects will be making calls, sending text messages, playing games, surfing the Internet, and interacting with friends on social media sites on this same device. These are ‘environmental risks’ of collecting data in an unsupervised setting and can cause data collection to be interrupted or delayed. However, this issue is not unique to BYOD studies; clinical trial sponsors have never been able to control a subject’s behavior when they are completing any PRO assessment remotely, be it on paper or any other data collection mode.

Storage, or the amount of space available on the device, also becomes an issue with devices that are not under the sponsor’s control. This is especially true when considering all the tasks for which the subject uses his or her mobile device (e.g., watching videos, taking pictures, playing music, and downloading other apps). In addition, the subject can delete the app at any time, even with captured and un-submitted data. However, as part of the inclusion criteria, potential subjects could be required to agree to not delete the study app. Some checks can be conducted by site staff and via online monitoring; however, these options are limited.

Compliance questions are also raised since the sponsor does not own the device and can’t force the subject to have notifications turned on. If the subject chooses to turn off notifications, these automated alerts to complete required data entry task do not work. Like the agreement to not delete the app as part of the inclusion criteria, subjects could be required not to turn off notifications, which are associated with high compliance [27, 28]. If the subject does turn off notifications during the trial and compliance with data entry is negatively impacted, the study staff’s review of online reports should alert them to a compliance problem.

6.7 Technology Changes

The set of mobile operating systems that support the mobile market are changing constantly and quickly. For sponsors of clinical trials, the pace with which change happens will become a major challenge. Agile software developers iterate through releases quickly, with some deploying new features on a monthly basis. Support for the previous version of the software is dropped as soon as the new app is made available because app stores only allow one version of the software to be sold at any time. Increases in the amount of infrastructure necessary to support the complex functionality of existing solutions and increased maintenance cycle costs for vendors may drive prices upward. The result may be that the cost-benefit ratio of switching to a BYOD solution is not as large as anticipated.

7 Discussion and Conclusions

The movement toward ePRO data collection has been one of the most significant advances in PRO measurement. Based on important advances in technology and in the infrastructure to enable the effective deployment of the technology, ePRO is quickly becoming the gold standard for PRO data capture in clinical trials. There is no doubt that electronic capture of PRO endpoint data is here to stay. Its numerous advantages over paper-based data collection result in more complete and accurate PRO datasets, which could be the difference between a failed and successful clinical trial. The significance of this has not been lost on the regulatory and scientific community. The FDA has asserted its support and expectations for electronic capture of clinical trial source data, including PRO endpoints [5, 19]. ISPOR has established three task forces that have issued ePRO-related good research practice recommendations [20, 21, 23]. In addition, due to the increasing importance of ePRO data collection in clinical trials, representatives of the pharmaceutical firms in the PRO Consortium (http://www.c-path.org/programs/pro) and ePRO system providers in the ePRO Consortium are working collaboratively on the development and implementation of ePRO data collection systems for all new PRO instruments being developed within the PRO Consortium. The future of ePRO data collection in clinical trials is bright and, although there remain a number of issues to be resolved, the promise of BYOD opens a new chapter in its evolution.